Determining earpiece presence at a user ear

ABSTRACT

In general, the subject matter described in this disclosure can be embodied in methods, systems, and computer-readable devices. An audio processing device plays a source audio signal with an electroacoustic transducer of a user earpiece, and records an aural signal that is sensed by same said electroacoustic transducer. The audio processing device determines values of one or more features of the aural signal that indicate a characteristic of a space in which the user earpiece is located. The audio processing device compares the determined values of the one or more features of the aural signal with pre-defined values of the one or more features. Based on a result of the comparing, the audio processing device determines whether the user earpiece is located at a user&#39;s ear.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims the benefit of U.S. Application Ser. No.62/379,160, filed Aug. 24, 2016, U.S. Application Ser. No. 62/342,872,filed May 27, 2016, U.S. Application Ser. No. 62/342,871, filed May 27,2016, and U.S. Application Ser. No. 62/342,869, filed May 27, 2016 thecontents of which Applications are incorporated herein by reference intheir entirety.

BACKGROUND

With the proliferation of mobile computing devices in recent years,users have turned increasingly to earbuds, headphones, and other typesof headsets to listen to an increasing supply of audio content madeavailable through these mobile computing devices. Smartphones, forinstance, typically include a headphone jack that allows users toconnect headsets to the phone, through which a user may listen to songsfrom a media library or streaming service, podcasts, audio tracks fromvideos, and a variety of other content sources of the user's preference.Users may also use headsets that include earbuds and an externalmicrophone placed near the user's mouth to hold telephone calls for ahands-free mobile call experience. While listening to audio through aheadset of a mobile device (or other computing device), the user may beinterrupted by various events that require the user's attention,including ceasing to listen to the audio for a period of time. In someinstances, the user may interact with applications or other resources ona device having restricted access. In some instances, the user mayattempt to listen to audio content in a noisy environment that makes theaudio content difficult for the user to hear.

SUMMARY

This specification generally relates to audio signal processing,including techniques for applying acoustic headsets connected to anaudio processing device (e.g., a phone) as a sensor for collecting datathat the device can leverage to provide a variety of services to a user.For example, the techniques discussed herein may enable a softwareplatform on a device to utilize off-the-shelf earphones as a powerful,external multi-sensor that can sense characteristics of a user and theuser's environment in real-time. Some implementations of the techniquesdiscussed herein include detecting the presence of an earpiece at auser's ear, verifying a person's identity based on acousticcharacteristics of a user's ear, performing active noise cancellationusing pre-recorded sounds, and calculating temperatures of variousenvironments using a transducer of an earpiece.

The details of one or more implementations are set forth in theaccompanying drawings and the description below. Other features,objects, and advantages, will be apparent from the description anddrawings, and from the claims.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 depicts a conceptual diagram of a user listening to a sourceaudio signal played through a smartphone, using a pair of earbuds. Asthe source audio signal is played, the earbuds simultaneously functionas a microphone to capture an aural signal that results within theuser's ear as the audio signal is played.

FIG. 2A is a block diagram of an example audio processing device that isconfigured to carry out various ones of the techniques described herein.

FIG. 2B is a schematic that illustrates a configuration between acoding/decoding component (commonly referred to as a “codec”) and anearpiece for simultaneous playing and recording using a transducer ofthe earpiece.

FIG. 2C is a schematic that illustrates another configuration between acoding/decoding component and an earpiece for simultaneous playing andrecording using a transducer of the earpiece.

FIG. 2D is a schematic that illustrates a configuration between acoding/decoding component and two earpieces for simultaneous playing andrecording using transducers of the earpieces.

FIG. 3 is a flowchart of an example process for authenticating a userbased on acoustic characteristics of a user's ear. The process involves(i) an enrollment phase in which an acoustic signature is generated fora specific user and registered with the user's account, and (ii) averification stage in which a user's identity is verified based oncomparison of features from an aural signal associated with a user toacoustic signatures that are registered with the user's account.

FIG. 4 is a flowchart of an example process for generating an acousticsignature of a user in the enrollment phase of the authenticationprocedure.

FIG. 5 is a flowchart of an example process for verifying a user'sidentity in the verification phase of the authentication procedure.

FIG. 6 is a flowchart of an example process for determining the impulseresponse of an acoustic echo (w_(a)) of a space in which an earpiece islocated, e.g., for generating an acoustic signature of a user, detectingthe presence of an earpiece at a user's ear, or authenticating a user.

FIG. 7 is a flowchart of an example process for detecting the presenceof an earpiece at a user's ear based on features derived from an auralsignal recorded during playing of a source audio signal.

FIG. 8 is a process for using a stored instance of a pre-recorded audiosignal to cancel interference resulting from the pre-recorded audiosignal playing in an environment of a user as the user listens to asource audio signal through a headset.

FIGS. 9A-D show a process for determining the temperature of a user andan environment using a transducer.

FIG. 10 shows another process for determining the temperature of a userand an environment using a transducer.

FIG. 11 depicts an example computer that may be configured to carry outthe computer-implemented methods and other techniques described herein.In some examples, an audio processing device may include a combinationof hardware and software like that discussed with respect to FIG. 2A.

Like numbers and indicators among the various drawings indicate likeelements.

DETAILED DESCRIPTION

This specification generally describes systems, methods, devices, andother techniques for processing audio signals. In some implementations,these techniques may be performed using earpieces (e.g., headphones orearbuds) by using a same electroacoustic transducer in the earpiece toboth output, as a speaker, a played audio signal and to sense, as amicrophone, an aural signal that can be recorded by an audio processingdevice. In some examples, an audio processing device can process anaudio signal sensed by an earpiece and compare that sensed audio signalto an audio signal that is played at the same time to determine whetherthe earpiece is currently located at a user's ear. In some examples, anaudio processing device can process an audio signal sensed by anearpiece to perform an authentication procedure in which a user'sidentity is verified based on acoustic characteristics of a user's ear.In some examples, an audio processing device may perform active noisecancellation by detecting a known audio signal that occurs in ambientnoise of an environment of a user, and by using a pre-recorded instanceof the detected audio signal to modify a source audio signal that a useris listening to through a headset.

Referring to FIG. 1, a conceptual diagram is shown of a user 102listening to a source audio signal s being played through earbuds 106.The source audio signal s is played by an audio processing device 104,which in this example is a smartphone. The earbuds 106 are lodged in theuser's ear 106 so as to form an acoustic chamber by which the user 102can listen to the source audio signal s. The source audio signal s maycarry any of a variety of audio content, such as music, speech, atelephone call, or an audio track from a movie, television show, orother video or media content.

As shown in the enlarged area 108, the earbud 106 is capable of bothoutputting the source audio signal s and sensing an aural audio signalr. In particular, the earbud 106 includes an electroacoustic transducerthat converts electrical signals corresponding to the source audiosignal s to soundwaves that can be heard by the user 102. By convertingthe electrical energy to acoustic energy (i.e., soundwaves), theelectroacoustic transducer functions as a speaker. However, theelectroacoustic transducer is also mechanically sensitive to acousticenergy in its environment, which causes the electroacoustic transducerto vibrate and thereby generate electrical signals. Accordingly, theelectroacoustic transducer is also capable of sensing an aural audiosignal r, and in some implementations may even output the audio sourcesignal s while simultaneously sensing the aural audio signal r. In someexamples, the electroacoustic transducer may include a diaphragm thatmoves back and forth to generate soundwaves due to pressure imparted onair located in front of the diaphragm. Electrical signals may drivemovement of the diaphragm using any of a variety of technologies such asby coils wound around electromagnets or piezoelectric crystals.

Generally, the aural audio signal r is the audio signal that results ina space in which the earbud 106 is located when a source audio signal sis played through the earbud 106. For example, if a microphone wereplaced adjacent to the earbud 106 while the earbud 106 lodged in theuser's ear played a source audio signal s, the signal captured by thatmicrophone would correspond to the aural audio signal r. Of course, theaural signal r would predominantly reflect the source audio signal splayed in the space, but it may further reflect other factors such asintrinsic characteristics of the earbud 106 and the unique acousticproperties of the user's ear that at last partially define the space inwhich the source audio signal s is played. It is noted that, forpurposes of illustration, the enlarged view 108 in FIG. 1 shows theearbud located externally of the user's ear, although in practice, theuser 102 would typically listen to an audio signal s with the earbud 106lodged within his or her ear.

In some implementations, the audio processing device 104 may beconfigured to both play the source audio signal s and to record theaural audio signal r as output and sensed, respectively, by the sameelectroacoustic transducer of earbud 106. A sound card or otherprocessing circuitry of the audio processing device 104 may beconfigured to output and record audio signals via a same electroacoustictransducer (e.g., of an earpiece that does include a dedicatedmicrophone distinct from a speaker of the earpiece). For example, anelectronic resistance component may be connected between one or moreconductors of a circuit that electrically interfaces an earpiecetransducer and an audio processing device. The placement of the resistormay form a voltage divider that provides a location in the electricalinterface from which an electrical signal corresponding to the auralaudio signal r can be sensed and, e.g., digitally sampled and recorded.Processing circuitry of the audio processing device 104 may drive anelectrical signal to the electroacoustic transducer for playing a sourceaudio signal s, while simultaneously recording the aural audio signal r.In some implementations, the processing circuitry may switch (e.g., at ahigh frequency) between alternate modes for playing a source audiosignal s and recording the aural audio signal r. The switching may occurat a sufficiently high frequency so that the user 102 does not perceiveany disruption in the playing of source audio signal s.

In some implementations, the earbuds 106 may be part of a premiumheadset that, for example, includes a dedicated microphone. Thededicated microphone may be used, for example, to record an aural audiosignal r or to record environmental noise, as may be done foractive-noise cancellation, for example. For instance, the dedicatedmicrophone may be an embedded microphone (e.g., 210 a, 210 b) or anexternal microphone may be employed (e.g., microphone 212).

Parameter Determination

Various ones of the techniques described herein involve determiningparameters associated with audio signals, earpieces, a space in whichthe earpiece is located (e.g., a user's ear canal), or a combination ofthese. Some of these parameters are reflected in the following model(Equation 1), which represents an aural audio signal r, as recorded byan audio processing device, as a function of these parameters.

r=s*(w0+T·wΔ+wa)+e+n   (Equation 1)

Note: the * operator denotes convolution

The parameters in the model of Equation 1 generally indicate thefollowing:

TABLE 1 r Digital samples recorded from aural audio signal sensed byearpiece s Digital samples of source audio signal to be played throughearpiece e Environmental noise that occurs in a space in which theearpiece is located (e.g., background or ambient noise) n Electricalnoise resulting from audio processing circuitry w0 Model of theelectrical echo of the earpiece at baseline temperature T₀ (e.g., inKelvins) wΔ Model of the temperature-dependent electrical echo of theearpiece wa Model of the acoustic echo of a space in which the earpieceis disposed T Temperature difference from T₀

Thus, the model of Equation 1 indicates that the recorded aural audiosignal r is a function of three audio signals: (1) source audio signal splayed by the audio processing device, (2) environmental noise e, and(3) electrical noise n of the audio processing device. However, thesource audio signal s may be distorted due to three factors, and thisdistortion is represented by the convolution of s with (w0+T·wΔ+wa). Theparameters w₀ and w_(Δ) are intrinsic parameters of the earpiece, andspecific to the design of a given earpiece. Thus, an earbud that fitswithin an ear may have different parameters w₀ and w_(Δ) values from anearphone that covers an ear. Likewise, a first earbud of a particularmodel by a particular manufacturer may have different w₀ and w_(Δ)parameters from a second earbud of a different model by a differentmanufacturer. These parameters generally indicate how the geometry andoverall configuration of an earpiece act to modify (e.g., distort) theaudio signal s when played through that earpiece. The w₀ parameterindicates how the earpiece modifies an audio signal sat a baselinetemperature T₀ (e.g., 20 degrees Celsius), while the w_(Δ) parameterindicates how the earpiece modifies an audio signal s as a function of atemperature difference from T₀.

In general, both w₀ and w_(Δ) represent a transformation characteristicof the audio signal s imparted by the earpiece independent of a space inwhich the earpiece is disposed. In some implementations, w₀ and w_(Δ)each comprise a plurality of values (e.g., tens or hundreds of values)that collectively indicate respective impulse response characteristicsof the earpiece. For example, w₀ can indicate the impulse responsefunction of the electrical echo of the earpiece, and w_(Δ) can indicatethe temperature-dependent impulse response function of the electricalecho of the earpiece.

A simplified way of describing the impulse response functions is toimagine the echo that results when an impulse tone is provided to theearpiece. For example, assuming that that the system samples at 1000 Hz,then the system would sample the amplitude of the earpiece voltage every0.001 seconds. If an impulse tone was output at time 0, the impulseresponse may record the echoes that the earpiece would hear and thatwere due to the output of the impulse tone, in this example beingrecorded as an amplitude measurement every 0.001 seconds. As such, theimpulse response may be considered a vector or array of values, onerecorded each 0.001 seconds after the playing of the impulse function.This echo can be determined to have a fixed characteristic that is basedon the characteristics of the earpiece (e.g., w₀) and a variablecharacteristic that is based on of the earpiece temperature (e.g.,T·wΔ).

The echo is also affected by an additional transformation characteristicthat represents the acoustic characteristics of the space in which theaudio signal is played, such as the user's ear canal (e.g., w_(a)),although this transformation characteristic is described in greaterdetail later and may be disregarded for certain measurements.Overlooking environmental noise and electrical noise for a moment, itcan be seen how the recording r taken using a transducer would be thecombination of sound s that was played using the transducer, convolutedwith multiple impulse responses that indicate how long it takes for thatsound to echo back to the transducer for recording (possibly appearingat multiple different times due to different echoes on differentsurfaces of the user's ear and the earpiece), and how loud that echo isat various times.

Returning now to discuss the transformation characteristics, theintrinsic parameters of the earpiece (w₀ and w_(Δ)) may be pre-definedand stored in a database that stores intrinsic parameters for aplurality of different earpieces. The audio processing device mayidentify these parameters from the database (e.g., from user inputindicating the make and model of his or her earpieces or based on anautomatic detection capability). In some implementations, the intrinsicparameters may be calibrated at a factory or laboratory and madeaccessible to a population of users. In some implementations, theintrinsic parameters may be calibrated by individual users with theirrespective audio processing devices.

In some implementations, the environmental noise signal e is determinedby recording the audio signal that results when the earpiece is locatedin position and no audio source signal s is played through the earpiece.In some implementations, the environmental noise signal e may berecorded from a signal sensed by a second microphone other than theelectroacoustic transducer of the earpiece, such as an externalmicrophone located in the environment of the user of the earpiece. Insome implementations, the environmental noise signal e is calculatedafter the system has determined the temperature T and the acoustic echoparameter w_(a), since the other variables and constants may be known oralready calculated (the circuit board noise N may either be known, orthe combination of the environmental noise signal e and the circuitboard noise N may be calculated in combination).

The acoustic echo parameter w_(a) may indicate an acousticcharacteristic of a space that the earpiece is located in when playingaudio source signal s. For instance, the w_(a) parameter indicates howthe physical space in which the earpiece is disposed imparts distortionon audio signal s, and is generally independent of how the earpieceitself imparts distortion on audio signal s. The size, shape,configuration, and material of the space in which the earpiece islocated may all affect the w_(a) parameter. Accordingly, when the sameearpiece plays a source audio signal s in different spaces, the sourceaudio signal s may be distorted in somewhat different ways based on theparticular characteristics of the respective spaces. The acoustic echoparameter w_(a) may be derived when various other values are known, suchas the audio source signal s, the aural audio signal r, and thetransformation characteristics w₀ and T·w_(Δ).

As is described in greater detail herein, the acoustic echo parameterw_(a) may be applied in different contexts, for example, to generateunique acoustic signatures for users based on the configurations ofusers' ears that form a space for an earpiece, to verify the identity ofusers, to detect whether an earpiece is disposed in the ear of a user,and to determine a temperature of an earpiece or a space within whichthe earpiece is located. In some implementations, the acoustic echoparameter w_(a) can comprise a plurality of values (e.g., tens orhundreds of values) that collectively indicate an impulse response ofthe space in which an earpiece is located. The impulse response functioncan thus model the transformation or distortion of audio signal sresulting from the acoustic characteristics of the space in which theearpiece is located.

In some implementations, an audio processing device may determine valuesfor the temperature T and the acoustic echo parameter w_(a) by solvingEquation 1 for the values, once other relevant parameters have beenidentified. An example process 600 for determining or solving for T andw_(a) using the above-described Equation 1 is represented in theflowchart of FIG. 6. In some implementations, the process 600 may beperformed by an audio processing device using conventional earpieces(e.g., earbuds or earphones) having an electroacoustic transducer tooutput and sense audio signals. To derive T and w_(a) for a given space,the earpiece is located in that space while an audio source signal s isplayed through the earpiece (stage 602) and while an aural signal r isrecorded from electrical signals generated by the earpiece transducer(stage 604). For example, a user could prompt determination of theacoustic echo parameter w_(a) for one of the user's ears by placing theearpiece in position at the user's ear and selecting a control on theaudio processing device that initiates performance of the process 600.In some cases, the values of w_(a) may depend in part on a position andorientation of the earpiece in a space, and so the user may take care tolocate the earpiece at or near a desired position in the space fordetermination of the w_(a) parameter. In some implementations, a spacemay be associated with multiple sets of values for the w_(a) parameter,each set corresponding to a different position of the earpiece in thespace.

At stage 606, the audio processing device identifies the intrinsicearpiece parameters w₀ and w_(Δ). These may be calibrated locally on theaudio processing device, stored on the audio processing device, orobtained from a computing system remote from the audio processing device(e.g., an Internet-based server system). The audio processing device mayidentify values for the intrinsic parameters that specifically apply tothe earpiece worn by the user. These values may not change based onposition of the earpiece or location of the earpiece in a space.

At stage 608, the audio processing device selects a first segment of theaudio source signal s and a corresponding segment of the aural signal rfor processing in an initial iteration. The selected segment of theaural signal r corresponds to the first segment of the audio sourcesignal s in that the selected segment of r indicates the aural signalthat occurred during a same period of time in which the first segment ofaudio signal s was played and output into the space by theelectroacoustic transducer of the earpiece. For example, if the systemis playing and recording audio at 1000 Hz, the selected r and s segmentsmay include the voltage of the played signal and the voltage of therecorded signal over a 0.001 second interval of time or recorded every0.001 seconds.

At stage 610, the audio processing device determines a source-reducedaural signal r_(diff). The source-reduced aural signal r_(diff) can becomputed by subtracting, from the selected segment of recorded auralsignal r, the convolution of the selected segment of the audio sourcesignal s and w₀ (i.e., rdiff=r−s*w0). As may be apparent from analyzingEquation 1, rdiff represents the portion of the recorded sound thatexcludes the w₀ effect of the earpiece on the recording.

At stage 612, the audio processing device determines thetemperature-dependent aural signal r_(Δ) as the convolution of theselected segment of the audio source signal s and w_(Δ) (i.e., rΔ=s*wΔ).As may also be apparent from analyzing Equation 1, r_(Δ) represents theportion of the recorded sound that excludes the w₀ effect of theearpiece on the recording.

At stage 614, the audio processing device can then determine atemperature T (either an absolute temperature T or a temperature thatrepresents an offset from the baseline temperature T₀). This temperatureT can be determined by identifying a value for T that minimizes thedifference between r_(diff) and the convolution of T and r_(Δ) (i.e.,T=argmin_(T)(r_(diff)−T*rΔ)). In some implementations, the value of Tthat minimizes this expression can be determined using a minimum leastsquares estimation technique. The identified value of T can be thedetermined temperature of the earpiece. As may be apparent fromanalyzing Equation 1, this identification of T does not account for theeffect on the recorded signal that w_(a), e, an d N may impart. This maybe because w_(a) may be assumed to be optimal and e and N may havenegligible effect on the identification of T during the initialdetermination of various coefficient values.

At stage 616, upon determining a value for T (stage 614), the audioprocessing device then determines values for the aural signal acousticecho r_(a). The aural signal acoustic echo is generally the component ofthe recorded aural signal r that results from audio played by theearpiece, and is the distortion imparted on the audio source signal s asa result of the acoustic characteristics of the the earpiece (e.g., notaccounting for w_(a), e, and N. The aural signal acoustic echo r_(a) canbe computed by identifying the recorded signal r, and subtracting fromit (1) the convolution of the selected segment of the audio sourcesignal s and w₀ and (2) the convolution of T, the selected segment ofthe audio source signal s and w_(Δ) (i.e., ra=r−s*w0−T*s*wΔ).

In practice, the values of T, and thus of r_(a), tend to change rapidly.An earpiece may warm up, for example, the longer it is lodged in auser's ear, or based on increasing intensity of the source audio signals during different segments of the song or as a result of the userincreasing volume for playing the source audio signal s. To account forthese rapidly changing values, the process 600 may determine a new valueof T and r_(a) for each of multiple small segments of the audio sourcesignal s and the recorded aural signal r before computing the values ofthe acoustic echo parameter w_(a). For example, T and r_(a) may becalculated for small segments of r and s (e.g., between 0.05 and 2seconds), and then w_(a) calculated after a longer time has passed(e.g., 2-10 seconds) based on values determined from multiple segments.These values may be averaged or otherwise combined at stage 622. In someimplementations, the values of w_(a) can be recalculated every 0.2seconds (i.e., the size of the interval of r_(a) every 0.2 seconds). Insome implementations these intervals for calculating w_(a) can be evensmaller, such as every 0.05 or 0.1 seconds. The values of r_(a)calculated at each interval can be combined by concatenation, i.e. bydividing r_(diff) into different segments, finding the correspondingvalue of T for each segment, calculating r_(a) and concatenating them.If additional segments of the source signal s and aural signal r remainto be processed before determining w_(a) (stage 622), then at stage 620the audio processing device selects the next segment of the signals sand r and returns to stage 610 for additional iterations. The process600 may continue these iterations until a threshold number of iterationshas passed or until another condition is satisfied.

At stage 622, the audio processing device then determines the value ofthe acoustic echo parameter w_(a). In some implementations, the value ofw_(a) can be computed as the value that minimizes the expression∥s*wa−ra∥. The values that minimizes this expression may be determinedusing various techniques, such as minimum least squares estimation. Asmay be apparent from analyzing Equation 1, the determination of theacoustic echo parameter w_(a) may not account for the values of e and N.

Once T and w_(a) have been calculated, e and N may be computed. Thesystem may wait a determined amount of time before performing such acomputation, in order to let the temperature of the earpiece level out(e.g., at least 5 or 10 seconds). At this point, all other values ofEquation 1 may be known, and the combination of e and N may becalculated. It may be possible to separate the value for e from thevalue for N if the system can determine characteristics of either signal(e.g., using another microphone to record a variation of e that iscommon to both recordings).

Referring back to FIG. 2A, a block diagram is shown of an example audioprocessing device 202 that may be configured to perform operations ofthe various methods described herein. The device 202 may be, forexample, a portable media player, a smartphone, a tablet computingdevice, a wearable computing device, a notebook or desktop computer, atelevision, or other types of computing devices that generally arecapable of playing, recording, and processing audio signals.

The device 202 includes an audio processor 220 having a player 222 forplaying audio source signals and a recorder 224 for recording signalssensed by a microphone 212. An earset 204 having a pair of earpieces 206a and 206 b (including respective electroacoustic transducers 208 a, 208b and optionally separate microphones 210 a, 210 b) may connect to theaudio processing device 202 via port 226 (e.g., a standard 3.5 mm audiojack). The earset 204 may be a conventional, off-the-shelf device thatdoes not have a dedicated microphone built into the earpiece. In someimplementations, the earset 204 may be a specialized device that, forexample, includes built-in microphones 210 a, 210 b. In someimplementations, the audio processing device 202 may include acontroller 218 that coordinates operations performed by the device; anauthentication engine 228 that performs aural-based authentication ofusers; an acoustic modeling engine 238 that generates values of acousticechoes w_(a) and acoustic signatures for users; an external noisedetector 236 for identifying ambient sounds and the occurrence ofpre-recorded audio signals in the ambient sounds; and a noisecancellation engine 240 for performing active noise cancellation usinginformation about pre-recorded audio signals identified by the externalnoise detector 236. The device may further include a communicationsinterface 242 that communicates over a network 216 with a server 214,e.g., to identify pre-recorded sounds and to obtain pre-stored models ofacoustic echoes w_(a). In some implementations, the audio processingdevice 202 may have fewer than all of these components, or may includeonly particular combinations or subsets of these components. As anillustration, in those examples in which the earphones are connected tothe audio processing device via a BLUETOOTH or other wirelessconnection, at least some of the audio processing may be performed bycircuitry at the headphones (e.g., the codec, the digital-to-analogoutput conversion, and the input analog-to-digital conversion may beperformed at the headphones). The external microphone 212 may also beconnected directly by permanent hard-wire connection to the audioprocessing circuitry in the headphones, rather than being connected tothe audio processing device 202 through a permanent hard-wire connection(e.g., not through a releasable 3.5mm audio jack). The details ofoperations performed by the audio processing device 202 are described inthe following sections.

Configuring Simultaneous Playing and Recording

A system can use a single transducer to both play and record audio. Byway of example, a transducer in an off-the-shelf earset may bepositioned so that it produces sound directed toward the ear canal, andso using that transducer also as a microphone may be useful given itsorientation toward the ear canal to receive sound, as will be understoodin view of intended applications for the technology described in thisdocument that are described below. The use of an earset transducer toboth play and record audio may be accomplished by either alternatingbetween the playing and recording functions, or by playing and recordingat the same time. In the former, the sound card output port and thesound card input port may be both connected to the transducer, and thesound card may alternate between outputting a signal to the transducerand recording a signal that is generated by the transducer, for example,by repeatedly playing audio for 200 mS and then recording audio for 10mS.

Regarding the second option, it is possible to both play and recordaudio at the same time. Doing so enables a computing system to analyzedifferences between the played audio and the recorded audio, which canbe helpful because it allows a computing system to determinecharacteristics of the environment in which the audio was played (e.g.,a temperature of the environment, sounds from the environment, andaudio-reflective characteristics of the environment, such as shape). Toenhance simultaneous playing and recording of audio, however, it ishelpful to modify the typical standard connection between the earpiecetransducer and the sound card (e.g., the sound card codec). Themodification can include inserting a resistor between the transducer andeither the audio circuit output or input. In effect, this insertion maybe the same as inserting the resistor between the audio circuit outputand input ports. This inserted resistor is illustrated in FIG. 2B asR_(ADDED). Before discussing the benefits provided by the addition ofthis resistor, this disclosure provides a brief, and somewhatsimplified, overview of transducer operation.

Referring now to FIG. 2B, the figure shows a codec 250 on the left,which represents the circuitry that can output an electrical signal to atransducer (e.g., through the V_(PLAYED) output port) and that caninput/analyze an electrical signal received from the transducer (e.g.,through the V_(RECORDED) input port). The codec may be located in amobile device (e.g., a phone) or may be located in the headset (e.g., inthe circuitry of Bluetooth-connected earphones). To the right of thefigure is an earpiece 260 which includes a transducer. In this example,both the V_(PLAYED) and V_(RECORDED) ports are connected to thetransducer which is being used both in a speaker configuration and amicrophone configuration. This contrasts with a typical operation whichwould involve one of the V_(RECORDED) port or V_(PLAYED) port beingconnected to the transducer by itself, depending whether the transducerwas to be used as a speaker or a microphone.

When the transducer is connected as a speaker to the V_(PLAYED) port ofthe audio circuitry, the transducer moves to create sound wavesaccording to the voltage that is output at V_(PLAYED.) Some discussionregarding the construction and functioning of a typical audio transducermay be helpful to explain how sound is created and recorded, and how theconfiguration described herein enhances simultaneous playing andrecording of audio. One of the main components of a typical transduceris the cone. This is the portion of the transducer that moves back andforth to create sound waves and is what is seen when viewing the frontof a speaker. It can be made of various flexible materials such aspaper, paper composites and laminates, or plastic materials. The fixedframe of the speaker within which the cone moves is called a basket, andthe cone is connected to the basket by a gasket around the edges, whichmay also be a paper, plastic, or rubber material. At the center of thecone is the coil, which is typically connected to the backside of thecone and is not visible when viewing a speaker from its front. The coilis connected to the cone and can move forward and backward with the conebecause the center of the cone at which the coil is connected may besuspended in space (the cone is only connected at its outercircumference). Surrounding the coil, but separated by air, is anelectromagnet that is fixed to frame. Applying voltage to theelectromagnet can induce an electrical field to cause the coil toattract to the electromagnet. Because the electromagnet may be displacedfrom the coil, inducing an electrical field may cause the coil to moveforward and backward. As such, applying voltage to the electromagnetaffects the position of the cone that is connected to the coil. Still, agiven voltage may not necessarily correspond to a fixed position of thecone, because the cone may have biasing forces imparted on it by thegasket and flexing properties of the cone. In practice, the voltage oraudio signal is typically oscillating to represent the variousfrequencies that output through movement of the transducer, and thehistory of voltages applied to the transducer affects its positioning.In this example, the voltage would be provided to the transducer by theaudio circuitry through the V_(PLAYED) port.

The position of the transducer, however, is not entirely set by thevoltage provided at the V_(PLAYED) port. Indeed, imagine a user pressingon the cone while the transducer is playing audio, or a shock wavehitting the transducer. In both these situations, the position of thecone deviates from that that would be expected without the any externalinfluence. Moreover, the movement of the cone due to any externalinfluences affect the voltage over the electromagnet. Indeed, just as agenerator works by moving a component with respect to an electromagnetto induce a voltage signal across that electromagnet, moving the coneand coil of a transducer can induce voltage and an electrical signalacross the terminals of an electromagnet. In fact, this is how thecircuit that is shown in FIG. 2B could function as a recording device—ifthe earpiece/transducer was connected only to the V_(RECORDED) port (notthe V_(PLAYED) port). In such an example, the pressure waves (e.g.,sounds) hitting the transducer would be converted to a voltage by theelectromagnet, which would be supplied to the V_(RECORDED) port toenable the codec to sample the changing voltage values at that port andoutput a digital data stream that represents the sound recorded by thetransducer.

As such, it can be seen how connecting the V_(PLAYED) or V_(RECORDED)ports to the transducer, at least one at a time, can enable playingaudio or recording audio using that transducer. Connecting both at thesame time to the transducer, however, may not be effective, because thenthe V_(PLAYED) and V_(RECORDED) may be shorted together and would bedriven to the same voltage (e.g., through connection with a conductor ofa resistance less than 0.1, 0.5, or 1 ohms). Introducing a resistorconnected between the V_(PLAYED) and V_(RECORDED) ports (e.g., with aresistance of at least 2, 5, 10, 50, or 100 ohms) and then connectingthe transducer directly to one of the V_(PLAYED) and V_(RECORDED) ports(e.g., again with a conductor of the above-described resistance values)allows voltages at the V_(PLAYED) and V_(RECORDED) to differ, enablingV_(RECORDED) to sense how the voltage across the transducer voltagediffers from that at V_(PLAYED). In this example, the placement of theresistor is illustrated as being placed between the V_(PLAYED) port andthe transducer, with the V_(RECORDED) port being directly connected tothe transducer, although alternatively the resistor could also be addedto the V_(RECORDED) line and V_(PLAYED) could be connected directly tothe transducer.

The addition of the R_(ADDED) resistor allows the voltage atV_(RECORDED) to differ from that present at V_(PLAYED). SinceV_(RECORDED) is connected between the transducer and ground, just aswith the transducer, V_(RECORDED) is thus able to sense the voltage ofthe transducer. For example, assume that 5V is present at V_(PLAYED) andthe voltage across the transducer would be expected to be 4V due to thevoltage division between R_(ADDED) and R_(TRANSDUCER.) Should thevoltage across the transducer differ from 4V due to movement of thetransducer that is influenced by external sounds, for example, with thevoltage being 4.1V, V_(RECORDED) will measure this voltage. A computingsystem is then able to determine that there was a 0.1 V deviation fromthe expected voltage 4V and the measured voltage at a moment in time.Regularly performing this determination, a computing system can extractthe relatively-small, but time-varying signal of the components of therecorded signal V_(RECORDED) that differ from the V_(PLAYED) signal.

In some examples, inserting R_(ADDED) into the circuit may notsignificantly affect the performance of the transducer, but may lowerthe amplitude of the voltage provided to the transducer or the voltagerecorded at V_(RECORDED), depending whether the resistor is placed on atthe V_(PLAYED) port or the V_(RECORDED) port, respectively. Thisreduction may similarly reduce the sound volume output by the transduceror the volume of the recording. As such, the computing system may beconfigured to switch the R_(ADDED) resistor in and out of the circuit,for example, only switching the resistor into the circuit when thecomputing system has been instructed to simultaneously play and recordaudio. When the resistor is switched out of the circuit, the earpieceport may be connected directly to the appropriate V_(PLAYED) orV_(RECORDED) port with a conductor of minimal resistance, as describedabove. The computing system may at the same time open a switch thatconnects V_(RECORDED) to the earpiece, so that V_(PLAYED) andV_(RECORDED) are not shorted together with R_(ADDED) removed.

In some examples, the V_(RECORDED) port may be a port that is typicallydesignated for dedicated microphones (e.g., the microphone that danglesfrom headphones on the cord intended to capture the user's voice). Assuch, the computing system may be configured to switch the V_(RECORDED)port to being connected to the same transducer as the V_(PLAYED) portwhen the computing system has been instructed to simultaneously play andrecord audio over a single transducer. As such, when headphones may bein their typical operation, the computing system may leave theV_(RECORDED) port open or connected to a different microphone. In suchexamples, the computing system is able to use a dedicated microphonewhen available or desired, and can use the same transducer as a speakeror microphone when a dedicated microphone is not available or when it isdesirable to use the same transducer for both playing and recordingaudio.

Referring now to FIG. 2C, the figure shows a schematic with an ADC(analog-to-digital converter) and DAC (digital-to-analog converter) tothe left, and a representation of a headphone for a user at the right.Connected between the two is a resistor R. This schematic is similar tothat shown in FIG. 2C, with the DAC representing V_(PLAYED), the ADCrepresenting V_(RECORDED), R representing R_(ADDED), and H representingR_(TRANSDUCER) (the ground connection of the codec is not represented inFIG. 2C). A primary difference from the schematic in FIG. 2B is that theADC includes a positive and a negative terminal that are connectedacross the resistor R, rather than the ADC connecting to just theearpiece side of resistor R. This configuration may be used in codecsthat include differential ADCs with positive and negative terminals, andmay represent a differential circuit in which the ADC is able to measurethe voltage difference over the resistor R. Such a configuration mayprovide increased dynamic range in comparison to the configuration thatis shown in FIG. 2B, in some implementations. The processing by thecodec or another computational device may be similar to that performedby the schematic in FIG. 2B. For example, the voltage over the resistorR may differ from that expected for a given output provided by the DAC,due to pressure waves changing the resistance of the headphone earpieceH. The codec (or another computational device) may be able to determinethe effect of these pressure waves on the headphone earpiece bycomparing the expected and actual measurements across the resistor R bythe ADC.

Referring now to FIG. 2D, the figure shows a schematic with two ADCs,two DACs, and two resistor Rs. This schematic is similar to thatpresented with respect to FIG. 2C, but shows a two-channelimplementation (left and right channels) rather than a one-channelillustration. In short, each channel has its own DAC outputting an audiosignal to its respective channel, its own resistor R between the DAC andthe corresponding input terminal to the audio jack that goes to the userearpiece (or the wire that goes to the corresponding transducer inwireless implementations that do not include audio jacks), and its ownADC to measure the differential across the resistor R. In this example,there is also an ADC for the microphone input, for example, for amicrophone that may be located inline the cord between the audio jackand the earpieces. This particular audio codec may have extra ADC inputsthat are available for use to measure the voltage across the resistorsR_(R) and R_(L). The ADC positive and negative inputs are reversed incomparison to those in FIG. 2C, but differential circuits may beconfigured with the positive and negative ACD inputs in differingconfigurations in circuits.

Aural-Based User Authentication

In some implementations, audio signals played and recorded through oneor more electroacoustic transducers of an earpiece may be used toimplement an authentication capability that can verify identities ofusers. Authentication can broadly be divided in two principal phases:(i) enrollment and (ii) verification. These phases are represented inthe flowchart of FIG. 3. During the enrollment phase (302), a userinteracts with an audio processing device to generate one or moreacoustic signatures that are registered with a user's account. Anacoustic signature is like a unique fingerprint associated with a user,e.g., resulting from unique geometries of the user's ear canal thatprovides distinct acoustic characteristics. During the verificationphase (304), the audio processing device determines whether features ofan aural signal at the user's ear matches an acoustic signatureassociated with the user (e.g., registered with a user's account), andtriggers different actions based on whether a match is determined thatverifies the user's identity.

For example, when a user arrives at a login page of a banking website,the user's computing device may automatically perform the auralauthentication procedure discussed herein to verify the user's identityin lieu of the user typing credentials into the page. If the user isauthenticated, the user may be permitted to log into a portal for abanking account associated with the user.

Aural-based authentication is not limited to single, discrete instancesof authentication, however (e.g., a one-time verification of a user'sidentity to determine whether to grant a user access to a restrictedsite). In some implementations, the aural techniques discussed hereincan be used to continuously and repeatedly authenticate a user overtime, e.g., during the course of a restricted activity. For example,while the user is logged into a portal for his or her personal bankingaccount, the user may wear a headset that locates a pair of earpieces atthe user's ears. As the user interacts with the banking site, the devicemay play and record sounds through the headset according to, e.g.,process 500 of FIG. 5, to monitor the user's status and continuouslyverify that an authorized user is present while the banking site is openon the device. If the user removes the headset, the authenticationprocedure can determine that the authorized user is no longer presentand can automatically cause the device to log out of the site, therebyensuring that a non-authorized user cannot hijack the authorized user'saccount. In some implementations, if authentication fails, the site mayprompt the user to confirm his or her identity or may automatically logthe user out of the site immediately or after a defined delay.

FIG. 4 depicts an example process 400 for enrolling a user on an audioprocessing device, including generating and registering an acousticsignature for the user.

At stage 402, the audio processing device receives input to initiateenrollment. In some implementations, the device may present a userinterface that prompts the user to enroll and provides step-by-stepinstructions for actions required of the user, such as instructions toinsert an earbud into the ear, adjust a position of the earbud, confirmthat a sound was heard, etc.

At stage 404, the audio processing device plays a source audio signal sthrough an earpiece. The audio processing device may, for example, drivea diaphragm of an electroacoustic transducer in the earpiece to causethe earpiece to generate soundwaves for the audio signal s. Varioustypes of audio content may be represented by the source audio signal s.In some implementations, the source audio signal s may be a familiarsong, podcast, or other audio track that the user has selected to listento at a given time. In some implementations, the source audio signal smay be white noise that would be audible to the user if the earpiecewere located at the user's ear. In some implementations, the sourceaudio signal s may be outside the normal range of human hearing (e.g.,above 20 kiloHertz) such that the audio source signal s cannot be heardby the user, even if the earpiece is located at the user's ear. Aninaudible source signal s may be beneficial, for example, to enableperformance of the earpiece detection procedure without disturbing theuser with unwanted sounds or in other environments where the user mayprefer silence or minimal noise. In some implementations, an inaudiblesource signal s can be added to an audible signal that a user hasalready selected to play on a device (e.g., music or other audiocontent) without being detected by a user. In some implementations, theaudio processing device may provide a physical or virtual interface(e.g., through a graphical user interface presented on an electronicdisplay of the audio processing device), which allows the user to selectthe type of audio signal s to play for in-ear detection (e.g., whitenoise, inaudible, or a pre-recorded audible sound signal). The audiosignal may be played for a relatively short span of time, e.g., 0.05-3seconds, or continuously.

At stage 406, the audio processing device records an aural audio signalr of sound that occurs at the user's ear where the earpiece is locatedwhile the audio source signal s is played. In some implementations,aural signal r can be recorded by the audio processing device based oninformation sensed by the same earpiece transducer that outputs theaudio source signal s. The audio processing device may simultaneouslyplay the audio source signal s and record aural audio signal r (e.g., bydriving the transducer and calculating a voltage of the transducer atthe same moment, or by the playing and recording being performedalternately from each other at high frequency in a time-division duplexscheme so that the user does not perceive that the playing is beinginterrupted by recording, such as alternating each no longer than each 1ms, 0.1 ms, or 0.01 ms). In some implementations, the audio sourcesignal s and aural audio signal r may actually be played and recordedconcurrently, without alternating between the stages. In someimplementations, the earpiece may include a second transducer separatefrom the first transducer, where the second transducer serves as adedicated microphone and the first transducer is a dedicated speaker.

At stage 408, the audio processing device determines values for theacoustic echo w_(a) (e.g., impulse response of acoustic echo ofearpiece) associated with the earpiece's current location during aperiod of time when the device plays the audio source signal s andrecords the aural signal r. The acoustic echo w_(a) determined at thisstage (408) indicates characteristics of the space in which the earpieceis currently located. An acoustic signature is then generated based onthe determined values for the acoustic echo w_(a). In someimplementations, the values of the acoustic echo w_(a) themselves formthe acoustic signature. In some implementations, the values of theacoustic echo may be further processed, filtered, and/or encrypted togenerate the acoustic signature for the user.

In some implementations, the audio processing device may generate duringthe enrollment phase a set of multiple (e.g., 5-10) acoustic signaturesfor a user. Different attributes may be assigned to different ones ofthe acoustic signatures that identify respective contexts of theacoustic signatures. For example, the attributes may indicate aparticular earpiece type, make, or model for the acoustic signature thatcorresponds to the earpiece that the user provides during enrollment.The attributes may further indicate an earpiece side (e.g., whether theearpiece is for the user's left or right ear) and a position of theearpiece at the ear. The audio processing device may, for example,prompt the user to rotate an earbud to different positions in the ear sothat a collection of possible signatures can be determined andassociated with the user, any of which may be valid during theverification phase depending on the position that the user later placesthe earbud in his or her ear. At stage 412, the acoustic signature isstored in association with an account of the user.

At stage 414, the audio processing device determines whether to promptthe user for signatures at additional in-ear positions. For example, theaudio processing device may require a pre-defined number n of acousticsignatures to be generated and registered for a user to completeenrollment, and the process 400 may continue until that number n ofvalid signatures is reached. If additional acoustic signatures arerequired, then the process 400 proceeds to stage 418 and the deviceprompts the user to adjust the earpiece to a new position. The process400 then repeats from stage 404 to generate an acoustic signature of theuser for the changed position of the earpiece. Once a sufficient set ofacoustic signatures has been generated, enrollment is completed (stage416).

Turning to FIG. 5, a flowchart is shown of an example process 500 forverifying a user's identity (i.e., authenticating a user). The process500 is generally performed after a user has enrolled and activatedaural-based authentication services on a device.

At stage 502, the audio processing device receives instruction toinitiate an authentication procedure. The instruction may be in the formof user input to the device, or may be an instruction from an operatingsystem or application on the device, e.g., an application that calls anAPI associated with an aural-based authentication service.

At stage 504, the audio processing device plays a source audio signal sthrough an earpiece. The audio processing device may, for example, drivea diaphragm of an electroacoustic transducer in the earpiece to causethe earpiece to generate soundwaves for the audio signal s. Varioustypes of audio content may be represented by the source audio signal s.In some implementations, the source audio signal s may be a familiarsong, podcast, or other audio track that the user has selected to listento at a given time. In some implementations, the source audio signal smay be white noise that would be audible to the user if the earpiecewere located at the user's ear. In some implementations, the sourceaudio signal s may be outside the normal range of human hearing (e.g.,above 20 kiloHertz) such that the audio source signal s cannot be heardby the user, even if the earpiece is located at the user's ear. Aninaudible source signal s may be beneficial, for example, to enableperformance of the earpiece detection procedure without disturbing theuser with unwanted sounds or in other environments where the user mayprefer silence or minimal noise. In some implementations, the audioprocessing device may provide a physical or virtual interface (e.g.,through a graphical user interface presented on an electronic display ofthe audio processing device), which allows the user to select the typeof audio signal s to play for in-ear detection (e.g., white noise,inaudible, or a pre-recorded audible sound signal). The audio signal maybe played for a relatively short span of time, e.g., 0.05-3 seconds, orcontinuously.

At stage 506, the audio processing device records an aural audio signalr that occurs at the user's ear where the earpiece is located while theaudio source signal s is played. In some implementations, aural signal rcan be recorded by the audio processing device based on informationsensed by the same earpiece transducer that outputs the audio sourcesignal s. The audio processing device may simultaneously play the audiosource signal s and record aural audio signal r as described above(e.g., with the audio source signal s and aural audio signal r beingplayed and recorded concurrently, without alternating between thestages). In some implementations, the earpiece may include a secondtransducer separate from the first transducer, where the secondtransducer serves as a dedicated microphone and the first transducer isa dedicated speaker.

At stage 508, the audio processing device determines values for theacoustic echo w_(a) (e.g., impulse response of acoustic echo ofearpiece) associated with the earpiece's current location during aperiod of time when the device plays the audio source signal s andrecords the aural signal r. The acoustic echo w_(a) determined at thisstage (508) indicates characteristics of the space in which the earpieceis currently located.

At stage 510, the audio processing device compares the values determinedfor the acoustic echo w_(a) determined at stage 508 with stored acousticsignatures associated with a user. If a match is identified (stage 512),then a specified activity may be permitted to be performed (stage 514),such as logging into a restricted account or unlocking a smartphone. Ifa match is not identified, then the specified activity may be denied ora different action taken by audio processing device (stage 514).

In some implementations, a device may store acoustic signatures formultiple different people that use the device. The authenticationprocess 500 can then be performed to identify a particular user amongthe multiple different users who have acoustic signatures registered onthe device. For example, a tablet computing device may be shared amongdifferent family members in a multi-person family. The device maydetermine who is interacting with the device based on an auralauthentication procedure in which the user places earphones on (e.g., tolisten to music, a podcast, a video soundtrack), and the deviceidentifies w_(a) values for the user and compares the w_(a) valuesagainst acoustic signatures associated with different ones of the familymembers. The comparison may reveal which of the family members iscurrently using the device based on, e.g., whose acoustic signature mostclosely matched the derived w_(a) values. In some implementations,content may be targeted to a specific user based on the auralidentification. For example, a video streaming application may haveaccounts or profiles associated with multiple people on a device. Thevideo streaming application may use aural-based verification toautomatically determine whose account or profile to open. In someimplementations, content suggestions or recommendations may be providedto a user based on identification of a given user on a device. Forexample, the device may maintain records of music or videos played bydifferent users and may correlate the records with respective useraccounts or respective acoustic signatures of users. When a person isverified, the device may access the records of media previously playedby that person and determine additional content to play or suggest tothat person. In some implementations, the verification phase ofaural-based on authentication can be performed invisibly to a user. Forexample, as a user listens to a media file through a headset, the devicemay continuously or at particular times perform a user verificationprocess to identify the user listening to the media file to collectinformation about the user's content preferences, to make personalizedcontent selections and recommendations to a user, or to otherwise adaptan experience on the device to the user's personal preferences.

Detection of Earpiece at a User's Ear

In some implementations, an audio processing device may be operable todetect whether an earpiece is located at a user's ear or is located awayfrom the user's ear. For an earbud that is configured to be lodgedwithin an ear, the audio processing device may determine when the earbudhas been inserted into the ear of a user or when the earbud is removedfrom the ear of a user. For an earphone (e.g., from a headphonesdevice), the audio processing device may determine when the earphone hasbeen placed over the ear of a user or has been moved off the ear of theuser. As described in the following paragraphs, the detected presence orabsence of an earpiece at a user's ear, or the detected change inlocation of the earpiece relative to the user's ear may triggerspecified actions to be performed by the audio processing device oranother computing device or system in communication with the audioprocessing device. For example if, as a user is listening to music oranother audio source signal s through his or her audio processing deviceand the user intentionally removes an earpiece from his or her ear todirect his or her attention to another activity, the audio processingdevice may detect that the earpiece has been removed from the ear andautomatically cause a media player application on the device to pausethe music. Later, when the user is ready to begin listening to the musicagain, the user may simply position the earpiece at the ear to cause thedevice to automatically resume playback of the music from the sameposition in the music track where the track had earlier been paused.

Referring to FIG. 7, the depicted flowchart shows an example process 700for determining the current location of an earpiece vis-à-vis a user'sear, and in particular, whether the earpiece is located at or away fromthe user's ear. This process may be able to specifically determinewhether an earpiece is located in a user's ear, in distinction to merelydetermining the presence of any object nearby the earpiece. In someimplementations, the process 700 may be carried out by the devices andsystems described herein, such as audio processing device 202 (FIG. 2A).

At stage 702, the audio processing device plays a source audio signal sthrough an earpiece. The audio processing device may, for example, drivea diaphragm of an electroacoustic transducer in the earpiece to causethe earpiece to generate soundwaves for the audio signal s. Varioustypes of audio content may be represented by the source audio signal s.In some implementations, the source audio signal s may be a familiarsong, podcast, or other audio track that the user has selected to listento at a given time. In some implementations, the source audio signal smay be white noise that would be audible to the user if the earpiecewere located at the user's ear. In some implementations, the sourceaudio signal s may be outside the normal range of human hearing (e.g.,above 20 kiloHertz) such that the audio source signal s cannot be heardby the user, even if the earpiece is located at the user's ear. Aninaudible source signal s may be beneficial, for example, to enableperformance of the earpiece detection procedure without disturbing theuser with unwanted sounds or in other environments where the user mayprefer silence or minimal noise. In some implementations, the audioprocessing device may provide a physical or virtual interface (e.g.,through a graphical user interface presented on an electronic display ofthe audio processing device), which allows the user to select the typeof audio signal s to play for in-ear detection (e.g., white noise,inaudible, or a pre-recorded audible sound signal). The audio signal maybe played for a relatively short span of time, e.g., 0.05-3 seconds, orcontinuously.

At stage 704, the audio processing device records an aural audio signalr of sound that results in a space where the earpiece is located whilethe audio source signal s is played. In some implementations, auralsignal r can be recorded by the audio processing device based oninformation sensed by the same earpiece transducer that outputs theaudio source signal s. The audio processing device may simultaneouslyplay the audio source signal s and record aural audio signal r, or theplaying and recording may be performed alternately from each other,e.g., in a time-division duplex scheme. In some implementations, theearpiece may include a second transducer separate from the firsttransducer, where the second transducer serves as a dedicated microphoneand the first transducer is a dedicated speaker.

At stage 706, the audio processing device determines values for theacoustic echo w_(a) associated with earpiece's current location during aperiod of time when the device plays the audio source signal s andrecords the aural signal r. The acoustic echo w_(a) determined at thisstage (706) indicates characteristics of the space in which the earpieceis currently located. Depending on whether the earpiece is currentlylocated at or away from the user's ear, the values of w_(a) may varysignificantly, thereby allowing determination of the location of theearpiece from the w_(a) values. In some implementations, the audioprocessing device determines the values of w_(a) according to theprocess 600 set forth in FIG. 6, as described previously.

At stage 708, the audio processing device accesses pre-stored values forthe acoustic echo w_(a)-_(stored) associated with earpieces that werepreviously determined to be located at an ear of a user. The values ofw_(a)-_(stored) are thus representative of values that would be expectedif an earpiece is located at the ear of the user. The values ofw_(a)-_(stored) may be determined based on analysis of the acoustic echofrom previous instances in which the earpiece was determined to belocated at (i) the ear of the same user for which the device determinedw_(a) at stage 706, (ii) the ears of other users (e.g., users or testersassociated with a provider of the in-ear detection service), or (iii)the ears of both the same user from stage 706 and other users. In someimplementations, the values of w_(a)-_(stored) may be automaticallyupdated based on updated values pushed to the audio processing devicefrom a server system or based on feedback from the user of the audioprocessing device indicating whether previous determinations of thelocation of an earpiece were accurate.

In some implementations, the audio processing device may accessdifferent values for w_(a)-_(stored) based on the particular earpieceworn by the user or based on a predicted position of the earpiece. Sincethe values of the acoustic echo parameter w_(a) may be sensitive todifferences among earpiece design and differences in position of theearpiece at the user's ear that impact the acoustic properties of thespace in which the earpiece is located, the audio processing device mayidentify the pre-stored values for w_(a)-_(stored) that most closelymatch the conditions of the user's earpiece during the course of process700. For example, the device may recognize that the earset connected tothe device are IPHONE earbuds, and may select values for w_(a)-_(stored)that are appropriately correlated with these types of earbuds. In someimplementations, the device may store information about the frequenciesat which a user has historically positioned an earpiece at his or herear, and may first access pre-_(stored) values for w_(a)-_(stored) thatcorrespond to one or more positions that the user most frequently placesthe earpiece at his or her ear. The values of w_(a)-_(stored) may differfrom acoustic echoes that were determined from earpieces that weremerely placed near non-ear objects or inside of non-ear objects. Inother words, the values of w_(a)-_(stored) may be specific to one ormore user ears. In some implementations the values of w_(a)-_(stored)may reflect acoustic echoes that were determined from earpieces thatwere placed near any objects (whether or not ear objects), such that theaudio processing device is able to determine whether the earpiece isnear any object.

At stage 710, the audio processing device compares the values of w_(a)(derived at stage 706) with the values of w_(a-stored) (derived at stage708). In some implementations, the device determines a similarity scorethat indicates how closely the values of w_(a) match the values ofw_(a)-_(stored). Generally, a closer match between the values indicatesa greater likelihood that the current location of the earpiece is at theuser's ear. At stage 712, the audio processing device may evaluate thesimilarity score with respect to a threshold score. If the similarityscore satisfies the threshold (e.g., is greater than or equal to thethreshold score), then the process 700 proceeds to stage 714 and thedevice outputs an indication that the earpiece is in the user's ear. Ifthe similarity score does not satisfy the threshold, the process 700proceeds to stage 718 and the device outputs an indication that theearpiece is not located in the user's ear. In some implementations, theaudio processing device compares one or more values for w_(a) to storedvalues of w_(a)-_(stored) or predetermined functions, thresholds,parameters, or tolerances, to determine whether the earpiece is near anyobject (not necessarily an ear). In response to determining that theearpiece is near an object or not near an object, the audio processingdevice may output an indication to that the earpiece is near an objector not near an object, to trigger a corresponding action.

In some implementations, different actions may be performed on the audioprocessing device or on another device based on whether the earpiece isdetermined to be located in the user's ear. For example, a first actionmay be performed based on identifying that the earpiece is located atthe user's ear (stage 716); alternatively, a second action may beperformed based on identifying that the earpiece is located away fromthe user's ear (stage 720). In some implementations, the audioprocessing device may continuously monitor the location of an earpiece,and an action may be triggered based on identifying that the earpiecehas been moved to a user's ear or identifying that the earpiece has beenmoved away from the user's ear.

Different types of actions that may be performed on the audio processingdevice or another computing device responsive to determining a locationof an earpiece with respect to a user's ear include playing a mediafile, pausing the media file, stopping the media file, resuming play ofa paused media file, activating an electronic display, adjusting abrightness setting of a backlight of the electronic display,transitioning a device into a locked mode, transitioning a device out ofa locked mode, initiating a telephone call, ending a telephone call,launching an application, or closing an application. For example, amedia player may pause a song if a user's earpiece falls out of theuser's ear; a telephone application may end a call, switch tospeakerphone or other internal speaker of the device, or generate analert to parties on a call if the user's earpiece is no longer detectedat the user's ear; or a media application may begin playing a song orother media file in response to determining that an earpiece has beenmoved to a location at the user's ear. In some implementations, forexample, when the device detects that the user has inserted the earpieceback into his or her ears, the device may revert to a mode that existedbefore the earpiece was removed, such as switching back fromspeakerphone to earpiece use during a telephone call.

Active Noise Control Using Pre-Recorded Sounds

In some implementations, an audio processing device may perform activenoise cancellation using pre-recorded sounds. Generally, thesetechniques allow a device to access a stored instance of a pre-recordedaudio signal and use the stored instance of the pre-recorded audiosignal to cancel interference or noise caused by the playing of anotherinstance of that same pre-recorded audio signal in an environment of auser. For example, a user may wear earphones during a workout at a gymto listen to songs on his or her personal playlist. The gym, however,may play (or even blast) music through a loudspeaker, and the presenceof the gym's loudspeaker music may make it difficult for the user tohear his or her preferred audio content. Accordingly, a microphoneassociated with the user's audio device (e.g., a microphone that is thesame electroacoustic transducer as the transducer that outputs a playedaudio signal, a separate microphone on the user's headset, a microphonethat is external to the audio device and separate from the user'sheadset, a microphone that is built into the user's headset, or two ormore of these microphones) may record an audio signal that identifiesenvironmental noise around the user, which includes the music playingthrough the gym's loudspeaker. The device may process that recordedaudio signal to identify the particular music playing through the gym'sloudspeaker (e.g,.a Lady Gaga song, a Rolling Stones song, or a Princesong), and then may access a copy of the same music and use that copy topredict in real-time how to cancel noise resulting from the song beingplayed through the loudspeaker in the user's environment.

One notable benefit of using pre-recorded sounds for active noisecancellation is that the latency in generating a modified,noise-cancelled audio signal can be substantially reduced as compared toother active-noise cancellation approaches. This is because the storedinstance of a pre-recorded audio signal can effectively serve as a mapthat the audio processing device can use to predict environmental soundbefore it even occurs.

An example process 800 for example noise cancellation with pre-recordedsounds is represented in the flowchart of FIG. 8. At stage 802, theaudio processing device plays a source audio signal s through anearpiece transducer located at a user's ear. For example, the user maybe listening to a podcast, but he or she may be located in a noisyenvironment such as a restaurant, coffee shop, or public transit. Atstage 804, the device captures an audio signal sensed by a microphoneconnected to the device (e.g., where the microphone could be the sametransducer through which the source audio signal s was played). Therecorded audio signal contains environmental noise (i.e., external orambient noise around a user). At stage 806, the device identifies apre-recorded audio signal p that occurs in the captured environmentalnoise. By way of example, the pre-recorded audio signal may be aparticular song played on the radio as background noise in a shop orrestaurant. The device may identify the particular song by accessing adatabase of pre-recorded audio signals and comparing a sample of theaudio signal p that was recorded from the microphone with portions ofthe audio signals stored in the database. At stage 808, the deviceaccesses a stored instance of the pre-recorded audio signal p′. Thedevice then synchronizes the stored instance of the pre-recorded audiosignal p′ with the audio signal p that occurs in the external noise,which can occur in real-time as the song continues to play in thebackground. At stage 512, the device cancels the pre-recorded audiosignal p from the audio source signal s using the pre-recorded audiosignal p′. For example, the pre-recorded audio signal p′ (or a processedsignal derived from the pre-recorded audio signal p′) may be subtractedin a manner that is synchronized with the background audio signal p, sothat the pre-recorded audio signal p′ cancels the distortion imparted onaudio signal s by background audio signal p. Additional techniques forperforming active noise cancellation are discussed in WIPO PublicationWO2014/178054, which is hereby incorporated by reference in itsentirety.

In some implementations, active-noise cancellation with a pre-recordedaudio signal can be performed as follows, using a common transducer inan earpiece to play and capture noise. The pre-recorded audio signal isrepresented as p. The signal recorded from the microphone (e.g., thecommon transducer in the earpiece) is represented as a. Anotherparameter used in this scenario is w_(room,) i.e., the room impulseresponse, which describes the relationship between p and a. Inparticular, a is the convolution of w_(room) and p. Note, this examplerefers to a “room” and a “room impulse response” to refer to theenvironment of a user, which is often an at least partially closedenvironment in which a loudspeaker outputs audio content.

The values of the w_(room) parameter may vary as the user moves aroundthe room and can vary depending on characteristics of the room itself inwhich the user is located. Generally, w_(room) is sensitive to theconfiguration of a given room where the user is located, the user'sposition in the room, the loudspeaker's position in the room, and therelative locations of the user (i.e. of the user's microphone) and theloudspeaker. If the location of the user and conditions in the room donot change, the values of w_(room) may remain generally static, but if,on the other hand, the location of the user or other conditions of theroom do change, the values of w_(room) may vary accordingly. Generally,for very small intervals of time (e.g., 0.01 seconds, 0.1 seconds),w_(room) will not change significantly from one interval to the next. Ifthe audio processing device has

As the user listens to a source audio signal s, and as one or moremicrophones (e.g., the electroacoustic transducer of the earpiece lodgedin the user's ear) sense audio signal a, the user's audio processingdevice may continuously and repeatedly re-calibrate w_(room). Once thedevice determines w_(room), and the device has identified thepre-recorded audio signal p from the environmental noise, the device canpredict a as the convolution of p and w_room for a next time interval(e.g., 0.1 seconds, while w_(room) remains relatively constant from thepreceding time interval). In some implementations, the audio processingdevice may re-calculate w_(room) and predict intervals for noisecancellation at even higher frequencies for less latency, e.g., every0.1-3 milliseconds. Once the device determines a, the device can performalgorithms to “inverse” the signal, such as finding a signal s that whenplayed from the headphones outputs—a, to cancel the background noise.

Temperature Measurement

Using the techniques described in this document, a device may measuretemperature using a transducer, for example, one that is located in anearpiece to play audio. The measured temperature may be that of theearpiece, but using various mechanisms described herein, the temperatureof a user's ear or of an environment in which the earpiece is locatedmay be inferred. There are various benefits to performing temperaturemeasurement using an earpiece rather than a dedicated thermometer. Amain benefit is that earpieces already on the market may be convertedinto temperature-sensing devices, without any modification of theearpieces being required in some implementations. As such, phones andother devices to which earpieces may be connected can be programmed tosense the temperature of a user or environment without adding adedicated temperature sensor. Moreover, the techniques described hereinare valuable because, even if one designed an earpiece or mobile deviceto include a temperature-specific sensor (e.g., a thermistor), the heatgenerated by the user, phone, or earpiece may affect the temperature ofthat temperature-specific sensor.

Enabling a mobile device to measure temperature with an earpiece servesmany beneficial uses. For example, health applications such as fitnesstrackers may regularly record a user's temperature and plot it overtime. As such, a user or computing system can compare recent usertemperatures to historical user temperatures to determine whether theuser may be falling ill or whether the user's physiology may bechanging, and may notify the user of such changes. Statistics may berecorded, for example, to better estimate calories burned by athletes,to alert athletes of possible hyperthermia during a workout, and toestimate days of female ovulation. Such mechanisms may also trackenvironmental temperature as a function of athletic performance (e.g.,by charting running times vs. outdoor temperature).

At least two example mechanisms can measure temperatures using atransducer. A first example mechanism is to estimate the temperature ofthe earphone (e.g., the transducer coil) itself, based on the effectthat earphone temperature has on the electrical properties of theearphone. This mechanism uses Equation 1 that is described previously inthis document to determine T at various points in time. Such a mechanismmay work for most or all earphones, so long as the system hascalibration information for that type of earphone. Still, it may take awhile for the earphone to heat or cool to the temperature of itsenvironment, so the mechanisms described herein can analyze theearphone's change in temperature to estimate the future earphonetemperature. The future earphone temperature can be compared tocalibration data to infer the current environmental and/or usertemperature. A second example mechanism is to analyze the recorded soundto determine properties of the air in a user's ear canal. Becauseproperties of the air such as temperature can affect the playing ofsound through that air, the temperature may be deduced by analyzing therecorded sound. This disclosure will next describe each of thesetemperature-determination mechanism.

First Mechanism: Measure Earpiece Temperature

In the first mechanism, as previously described, the system measurestemperature by simultaneously playing and recording audio (audible orinaudible) and detecting changes in the earphone's electrical propertiesdue to temperature changes of the earphone (e.g., its coil). Thissituation is similar to that of an electrical circuit that contains aresistor whose resistance is temperature-correlated, and where one partof the circuit is connected to a fixed output voltage (e.g., a soundcard output), and another part of the circuit is connected to a voltagemeter (e.g., a sound card input). With this configuration, one is ableto determine the current impulse response of the circuit. The currentimpulse response can be compared to previously-stored,temperature-dependent impulse responses to determine the currenttemperature of the resistor. This process is described in detail withrespect to certain portions of the process illustrated with respect toprocess 900 (see FIGS. 9A-D).

At box 902, the computing system determines T by recording a number ofaudio samples rand determining how temperature affected r given theplayed audio samples s and known transformation characteristics of theearpiece, such as w₀ and w_(Δ), that affect the played audio signal.Doing so involves use of Equation 1.

At box 904, the operations of box 902 can be performed by using at leasta portion of process 600 (see FIG. 6). In particular, the computingsystem can perform steps 602-624 of process 600 to determine T.

At box 906, the computing system performs a variation of process 600, bysolving for T, but adding a high-frequency noise to the output audiosignal s. Adding this high-frequency signal to s may be helpful becauses may not otherwise include audio at certain times, and becausetemperature measurement may be more effective with high-frequencysignals. Modifying s in this manner can involve performing the process600 with some variations thereto, which are described below with respectto boxes 908-914.

At box 908, the computing system may set s=s₀+s_(highfreqs). An examplehigh frequency signal may be a 20,000 Hz sine wave.

At box 910, the computing system may perform the operation of box 610,by calculating r_(diff)=r−s*w₀, but with the s value that has beenmodified to include a high-frequency value, as described above.

At box 912, the computing system may perform the operations of box 612,by calculating r_(Δ)=s_(highfreqs)*w_(Δ). In this example, the operationsubstitutes s_(highfreqs) instead of s, because s_(highfreqs) may besuitable for calculating temperature (although s=s₀+s_(highfreqs) may beused for other calculations, such as determinations of w_(a)).s_(highfreqs) may not be audible, and thus may not significantly affectw_(a), but still in some examples, s_(highfreqs) is not added to s fordeterminations of w_(a).

At box 914, the computing system may perform the operations of box 614by determining the temperature T that minimizes the difference betweenr_(diff) and the convolution of T and r_(Δ) (i.e.,T=argmin_(T)(r_(diff)−T*rΔ)). The values of r_(diff) and r_(Δ), however,may have been influenced by the addition of s_(highfreqs).

At box 916, the computing system may combine multiple values for T overa determined time period. For example, the system may concatenatetemperature measurements determined over a time period that is 0.03,0.05, 0.1, 0.2, or 0.5 seconds long. Example types of combinationinclude averaging and linear regression.

At box 920, the computing system determines the absolute temperatureusing the proportional temperature T that was determined at box 916.This determination may be performed because the value T may not providean absolute temperature, but may simply be a voltage value that changesbased on temperature, and that can be converted to an absolutetemperature (e.g., centigrade) by providing an offset and scalingconstant. Since there are multiple temperature described values here,the absolute temperature identified at box 916 will be denoted T_(c),and the relative temperature will be denoted as T_(b). Combining thesevalues into an equation with β to identify an offset and α to representscaling provides T_(b)=T_(c)α+β.

At box 922, the computing system (or another system) calibrates thevalues for α and β. This calibration may be performed ahead of time(e.g., before any temperature measurements are taken). For example, thecalibration may be performed when a user initially uses a particularearpiece, or can be performed by the manufacturer of the earpiece or byanother third-party organization. In these later examples, thecalibration values may be shipped with a mobile device or withheadphones, or may be accessible through an internet database. There areat last six mechanisms to determine these calibration values, describedbelow with respect to boxes 924-934.

At box 924, in a first mechanism, a user wears the earpiece for adetermined amount of time (e.g., 5 or 10 minutes), and the earpiecetemperature after that period of time is assumed to be a typical userbody temperature. For example, the earpiece temperature may be assumedor estimated to be a mean body temperature of a human (e.g., 36.7degrees). There may be no measurement of user temperature. Rather, theremay be just an assumption that the earpiece is the average humantemperature. In some examples, the temperature is estimated to be atypical temperature of an earpiece when placed in an ear of a user witha mean body temperature in a room of a mean room temperature.

At box 926, in a second mechanism, the user wears the earpiece for thedetermined period of time, and the system asks the user to inputtemperature T_(c), which may be the user's temperature or the meantemperature of the earpiece and the ambient temperature.

At box 928, in a third mechanism, the earpiece is placed in a room and,after a determined amount of time, the user is asked to input thetemperature of the room.

At box 930, in a fourth mechanism, the temperature of the earpiece ismeasured multiple times at different times of a day and/or on differentdays. The mean or median temperature measurement in such an example maydesignated as T_(b) while the mean or median temperature at locations ofthe mobile device and earpiece at such times according to weatherhistory data may be designated as T_(c).

At box 932, in a fifth mechanism, the earpiece may be placed in a roomor other environment, and the system may be connected to an externalsensor that determines the absolute temperature T_(c) of the environmentin which the earpiece is place, while the earpiece is used to measureits relative temperature T_(b).

At box 934, in a sixth mechanism, a second earpiece may be calibratedbased on an already-calibrated first earpiece. First, a first earpiecemay be calibrated using any of the above-described mechanisms. Second,the system may then set w_(Δ)=w_(Δ) (alpha) (e.g., by multiplying w_(Δ)by a coefficient and setting it as the new w_(Δ)). Third, some featureof w_(Δ) may be computed and denoted as C (e.g., 2 norm, C=∥w_(Δ)∥ orsome weighting on the frequency response of w_(Δ), or some fractionbetween the response of the earphone in two frequencies). Fourth, a usermay connect a second earpiece to a device and the device may compute thesame feature of w_(Δ) and denote that feature as D. Fifth, the devicemay calculate the w_(Δ) of the second earpiece given C, D, and the w_(Δ)of the first earpiece using the following formula: w_(Δ-second) _(_)_(earpiece)=C/D w_(Δ-first) _(_) _(earpiece). A similar mechanism usingw₀ instead of w_(Δ)also applies.

Faster Prediction of Temperature

Sometimes, an earpiece's heat transfer coefficient is small, which meansthat the earpiece coil and membrane sometimes take a while to reach thetemperature of its environment. As an example, in some earpieces, thetime constant is around 100 seconds, which means that if the earphone istaken out of a pocket at 30 degrees centigrade and is placed in a 15degree environment, it could take more than five minutes for theearphone to achieve 15 degrees, plus or minus 0.5 degrees.

At box 940, the computing system identifies the steady-state temperatureof the earpiece. This is the temperature that the earpiece may level outat after being in an environment for a certain amount of time. In otherwords, the earpiece may have the same temperature as its environment atsteady state.

At box 942, the computing system may wait a determined amount of time(e.g., 5 minutes in the above example) before designating thetemperature of the earpiece as the steady-state earpiece temperature.Essentially, the computing system may wait for the earpiece temperatureto heat up or cool down before designating the measured temperature asthe steady-state temperature. In some examples, this determination ofthe amount of time involves the computing system determining when thetemperature has stopped changing or that the rate of change intemperature has fallen below a determined threshold.

At box 944, the computing system may use a model of calibratedtemperature change over time for the earpiece—along with recent changesin earpiece temperature—to estimate the temperature of the earpiece inthe future (e.g., at steady state), even though the earpiece temperaturemay still be changing and has yet to reach steady state. Using thismodel may enable estimating the earpiece steady state more quickly thanwaiting minutes for the earpiece to heat up or cool down. Four suchmodels are described below, with reference to boxes 946 through 952.Because the earpiece may be have a complex body with a coil, plastic, amembrane, and potentially other features, Newton's law of cooling mayprovide a poor estimation of the earphones temperature in somescenarios, because Newton's law of cooling may best apply to simplematerials, such as resistance temperature detectors.

At box 946, as a first example, Newton's law of cooling may be used todetermine the future temperature of the transducer, for example, inscenarios in which Newton's law of cooling would provide an adequateestimation of the earphone's temperature.

T(t)=T _(env)+(T(t ₀)−T _(env))e ^(−r(t−t) ⁰ ⁾

In this equation, e is the temperature at time t, r is the heat transfercoefficient that depends on the type of earpiece, and T_(env) is thetemperature of the environment. With the goal of reading temperaturesamples to estimate T_(env), a system can predict the temperature when ris uncalibrated using the following four steps: (1) Use T₁, T₂, . . .T_(n) at times t₁, t₂, . . . t_(n) as inputs; (2) Use the approximation

$\begin{matrix}{{T_{(t)}^{\prime} = \frac{T_{i} - T_{i - 1}}{t_{i} - t_{i - 1}}};} & (3)\end{matrix}$

Use the equation T′^((t))=−r(T_(env)−T(t)) to solve the MLa,b=argmin_(a,b)Σ_(i) (T′_(i)−(a+bT_(i)))²;and (4) set r=b,

$T_{env} = {- {\frac{a}{b}.}}$

If r is calibrated, the following two steps may be performed: (1) UseT₁, T₂, . . . T_(n) at times t₁, t₂, . . . t_(n) as inputs; and (2)Solve the MLSQ of the following:

$T_{env} = {{argmin}_{T_{env}}{\sum\limits_{i}\; \left( {T_{i} - \left( {T_{env} + {\left( {{T\left( t_{0} \right)} - T_{env}} \right)e^{- {r{({t_{i} - t_{0}})}}}}} \right)} \right)^{2}}}$

This second set of operations may be more quick to perform than thefirst set of operations because r has already been calibrated. Thus,once r has been calibrated, the system may perform the second set ofoperations without recalibrating r.

At box 948, as a second example, a General Model may be used todetermine the temperature at a time in the future. There existsdecreasing functions F₊, F−>0, such that, (F₊, F−→0)

if T(t ₀)<T _(env) :T(t+t ₀)=T _(env) −F ₊ ⁻¹(t+F ₊(F+(t ₀)−T _(env)))

if T(t ₀)>T _(env) :T(t+t ₀)=T _(env) +F ⁻ ⁻¹(t+F ⁻(F ₊(t ₀)−T _(env)))

The functions F₊, F⁻ depend on the earphone's model and material. Theseequations may be solved using a linear least squares algorithm.

At box 950, as a third example, a General Linear Model may be used topredict the temperature at a time in the future, using a function F:

T′(t)=F(T _(env) −T(t))

With this in mind, temperature prediction may be performed using aninput of temperature measurements T₁, T₂, . . . T_(n) at times t₁,t₂, .. . t_(n) and a calibrated e. The system can then solve the followingoptimization problem using enumeration, binary search, or some othermethod:

T _(env)=argmin_(T) _(env) (T′ _(i) −T(t))²

At box 952, as a fourth example, a Sum of Exponents modeling may be usedto predict the temperature at a time in the future:

T(t)=T _(env)+sum_(i=1 . . . k) [a _(i) e ^(−r) ^(i(t−t0) ⁾]

In this example, the constants a_(i) and r_(i) are coefficients thatdepend on the earphone's model. This equation may be solved using alinear least squares algorithm.

In some implementations, although the example shown in the figurespresent calculation of the steady-state temperature after the computingsystem has converted the relative temperature to an absolutetemperature, the system may calculate the steady-state temperaturebefore the temperature is converted from a relative-temperature to anabsolute temperature. The conversion may then occur after thecalculation of the steady-state temperature.

Body Temperature Measurement (First Approach)

Once the system has measured the temperature of the earpiece (e.g.,either by waiting a determined amount of time (see box 942), or usingthe above-described temperature models to calculate future earpiecetemperature (see box 944)), a user ear/body temperature, such as theirtympanic temperature may be determined. The earpiece coil may be spacedapart from the user's tympanic membrane, and therefore may have atemperature that is based on a combination of the user's ear temperatureand the environmental temperature.

This disclosure describes mechanisms to determine user temperature andeven environmental temperature when the temperature of an earpiece isknown. In fact, although the following description is explained withreference to a transducer that is used as both a speaker and amicrophone, the process would also apply to body and environmenttemperature determinations that use other types of temperature-sensingdevices, such as dedicated microphones, thermometers, resistancetemperature detectors, and thermistors. This is because the belowdiscussion explains how to determine a user or environmental temperaturesometimes using just the temperature-sensing device (which just happensto be a transducer in this example).

More specifically, and as just mentioned, the temperature of theearpiece (or another temperature-sensing device) depends on the ambienttemperature and the temperature of the user's body, such as the earcanal temperature. As such, determination of the body temperature maysometimes involve obtaining multiple temperature measurements over time.Still, the temperature curve that can be constructed from the multipletemperature measurements may be unique for a given ambient/environmentaltemperature. Accordingly, using the shape of the temperature curve, thesystem may be able to determine the temperature of the user and theambient temperature. This process includes four basic steps, which arediscussed in turn.

At box 960, the computing system can calibrate the behavior of thetemperature curve in multiple ambient temperatures.

At box 962, the computing system can calibrate the ear temperatureversus the earpiece temperature in many different ambient temperatures.For example, the system may store, for each of multiple earpiecetemperatures, multiple ambient temperatures and the user temperaturesthat correspond to that specific set of earpiece and ambienttemperatures. In other words, the system may store many sets of threetemperature, or otherwise use such information to interpolateintermediate values using predictive formulas.

Both these calibrating steps may be performed ahead of time, forexample, before any real-time temperature measurement is taken. Indeed,the calibration may occur at a factory and be pre-installed with theearpiece or mobile device, or may be accessible through an internetdatabase.

At box 964, the computing system may record multiple earpiecetemperature measurements.

At box 966, the computing system can use the multiple earpiecetemperature measurements and the calibration information identified atbox 960 to determine the ambient temperature.

At box 968, the computing system can use one or more of the multipleearpiece temperature measurements and the calibration informationidentified with respect to box 962 to determine the user temperature.For example, the system may identify the user temperature that matchesthe determined earpiece temperature and the determined ambienttemperature.

There are at least two models that can be used to calculate the ambientand user temperatures given a series of temperature measurements andcalibrated values.

The first such model is a general model. There exist decreasingfunctions F₊, F⁻, G, G₊>0, such that (F₊, F⁻, G⁻, G₊→0) ifT(t₀)<αT_(ear)+(1−α)T_(amb) The functions F+, F− depend on variouscharacteristics, such as the earpiece model, material, and design.

T(t+t ₀)=αT _(ear)(1−α)T _(amb) −F ₊(t+F ₊ ⁻¹(F ₊(t ₀)−T _(ear)))−G₊(t+G ₊ ⁻¹(G ₊(t ₀)−T _(amb)))

if T(t ₀)<αT _(ear)+(1−α)T _(amb) , T(t+t ₀)<αT _(ear)(1−α)T _(amb) +F⁻(t+F ⁻ ⁻¹(F−(t ₀)−T _(ear)))+G ⁻(t+G ⁻ ⁻¹(G ⁻(t ₀)−T _(amp)))

To solve for the general model, the system can first calibrate for F andG for various values of ambient temperature and a. Then, a system maydeduce T_(amb) and T_(ear) using MLSQ.

The second such model is Newton's model.

T(t)=P[T _(amb) ,T _(ear)]+(T(t ₀)−T _(env))e ^(−R[T) ^(amb) ^(](t−t) ⁰⁾

Sometimes, P[T _(amb) ,T _(ear) ]=αT _(amb)+(1−α)T _(ear), and thenT(t)=αT _(amb)+(1−α)T _(ear)]+(T(t ₀)−T _(env))e ^(−R[T) ^(amb) ^(](t−t)⁰ ⁾

In this example, P(T_(amb,) T_(ear)) is the final temperature of theearpiece when the ambient temperature is T_(amb) and the eardrumtemperature is T_(env.) R[T_(amb)] is a function using the ambienttemperature. In this model, the heat coefficient may depend on theambient temperature. To solve for Newton's model the system may performfive steps. First, the system may calibrate R(T_(amb)) for some valuesof ambient temperature, and P. Second, the system may interpolateR(T_(amb)) (e.g., by plotting a straight line between at least twopoints to enable estimating points in between or to the sides of the twopoints). Third, the system may deduce r by continuously recording theearpiece temperature. Fourth, the system may find T_(amb,) such thatR(T_(amb))=r, since T_(amb) may be known, and T_(inferred) may bemeasured (either by waiting some time, or using the above-describedtime-prediction techniques). Fifth, using the above-described equation,T_(ear) may be determined by solving the following equation for T_(ear),since P may be known, T_(amb) may be known, and T_(inferred) may beknown (using prediction):T_(inferred)=P(T_(amb), T_(amb))

Body Temperature Measurement (Second Approach)

In this alternative approach, the computing system calculates theearpiece temperature as described previously, but the earpiecetemperature is designated as the ambient temperature due to the earpiecebeing determined to be located outside of the user's ear and thusinfluenced primarily by the ambient temperature. (The in-ear-detectiontechnique described with respect to process 700 is used to identify whenthe earpiece is outside of the ear.) Once the ear piece is determined tobe in the ear, the system may be able to determine the user temperaturesince the ambient temperature is known and the earpiece temperature isknown, and the system may have access to previously-performedcalibrations that map ear temperature to earpiece temperature forvarious ambient temperatures. This process is described in more detailwith respect to boxes 980-986.

At box 980, the computing system determines whether the earpiece is inthe user's ear, for example, using the in-ear-detection process 700. Thecomputing system may regularly perform this process, until the systemdetermines that the earpiece is not in the user's ear, at which pointthe computing system may perform the operations of box 982 toeffectively begin the temperature-measurement process.

At box 982, the computing system may determine the absolute steady-stateearpiece temperature, and designate this temperature as the ambienttemperature. The determination of the absolute steady-state earpiecetemperature may involve performing the operations of boxes 900-940 tocalculate the steady-state temperature of the earpiece. In this case,since the earpiece is not in a user's ear, it can be assumed that theearpiece is the same or at least similar to the ambient temperature ofthe space in which the earpiece is located. As described with respect tobox 940, determining the steady-state temperature can involve waiting adetermined length of time until the earpiece reaches the ambienttemperature (box 942), or can involve using a model to calculate thefuture steady-state temperature (box 944).

At box 984, the computing system determines whether the earpiece is inthe user's ear, for example, using the in-ear-detection process 700. Ifthe earpiece is not determined to be in the user's ear, the systemcontinues determining the absolute steady-state earpiece temperature(box 982). If the system determines that the earpiece is now located inthe user's ear, the computing system performs the operations of box 966.In various examples, the computing system continues to record theearpiece temperatures and does not perform the operations of box 982 todetermine the steady-state earpiece temperature until the computingsystem has determined whether the earpiece has transitioned to being ina user's ear. Waiting to determine the ambient temperature allows thecomputing system to select between simply using the last temperaturereading (e.g., because the earpiece has reached steady-stateenvironmental temperature) or using temperature modeling (e.g., becausethe earpiece temperature was still changing).

At box 986, the computing system calibrates ear temperature versusearpiece temperature for multiple ambient temperatures (e.g., eartemperature versus ambient temperature for multiple earpiecetemperatures), as described previously with respect to box 962.

At box 990, the computing system determines the ear temperature usingthe absolute steady-state earpiece temperature, the determined ambienttemperature, and the data that calibrated ear to earpiece to ambienttemperatures. For example, the computing system may identify an earpiecetemperature from the calibrated data that matches the determinedearpiece temperature, then from among multiple ambient temperaturesstored for that earpiece temperature, select the ambient temperaturethat matches the stored ambient temperature, and with these two valuesalready selected, select the corresponding ear temperature from the dataset. In some examples, the calibrated data uses interpolation or asimilar process where the recorded values are not identical to thecalibrated data.

Second Mechanism: Measure Characteristics of Environment

As the temperature changes, the acoustic characteristics of anenvironment in which the earpiece is located may change. For example,temperature can affect the speed of sound in an environment, as well asthe amplitude and other characteristics. As such, the acoustic echor_(a) and acoustic response w_(a), which are affected by sound speed andamplitude, may change as the temperature changes. Accordingly, comparingeither the acoustic echo r_(a) or acoustic response w_(a) tocorresponding values that are calibrated for different temperatures, canallow a system to determine the temperature inside a user's ear. Thisprocess is described below with respect to process 1000 (FIG. 10) andboxes 1002 through 1008.

At box 1002, the computing system extracts the acoustic echo r_(a) orthe acoustic response w_(a) while audio is being played, as describedpreviously with respect to process 600.

At box 1004, the computing system accesses a set of pre-calibratedacoustic echoes or responses, where there is at least one pre-calibratedvalue for each temperature value.

At box 1006, the computing system compares the extracted acoustic echor_(a) or the acoustic response w_(a) to a pre-calibrated acoustic echoor response P_(a) that was calibrated for an estimated temperature,designated here as X degrees Celsius.

At box 1008, the computing system uses a feature from the extractedacoustic echo r_(a) or the acoustic response w_(a) and compares thatfeature to a same feature of the pre-calibrated acoustic echo orresponse P_(a), to identify a matching ear canal temperature. As a firstexample, the system may extract the time difference between the playedand recorded signal, which may be temperature correlated according tothe effect of temperature on the speed of sound. As a second example,the system may compare the amplitude of the recorded signal to theamplitude of the played signal. As another example, the system may findthe entire impulse response between the played and recorded signal,compare it to the impulse response at X degrees, and use a model ofcontraction due to the speed of sound. Such mechanisms may be used todetermine a relative temperature of the ear.

At box 1010, the computing system converts the determined relativetemperature of the ear to an absolute temperature of the ear, forexample, by performing the operations of box 920.

FIG. 11 is a schematic diagram of a computer system 1100. The system1100 can be used to carry out the operations described in associationwith any of the computer-implemented methods described previously,according to one implementation. The system 1100 is intended to includevarious forms of digital computers, such as laptops, desktops,workstations, personal digital assistants, servers, blade servers,mainframes, and other appropriate computers. The system 1100 can alsoinclude mobile devices, such as personal digital assistants, cellulartelephones, smartphones, and other similar computing devices.Additionally the system can include portable storage media, such as,Universal Serial Bus (USB) flash drives. For example, the USB flashdrives may store operating systems and other applications. The USB flashdrives can include input/output components, such as a wirelesstransmitter or USB connector that may be inserted into a USB port ofanother computing device.

The system 1100 includes a processor 1110, a memory 1120, a storagedevice 1130, and an input/output device 1140. Each of the components1110, 1120, 1130, and 1140 are interconnected using a system bus 1150.The processor 1110 is capable of processing instructions for executionwithin the system 1100. The processor may be designed using any of anumber of architectures. For example, the processor 1110 may be a CISC(Complex Instruction Set Computers) processor, a RISC (ReducedInstruction Set Computer) processor, or a MISC (Minimal Instruction SetComputer) processor.

In one implementation, the processor 1110 is a single-threadedprocessor. In another implementation, the processor 1110 is amulti-threaded processor. The processor 1110 is capable of processinginstructions stored in the memory 1120 or on the storage device 1130 todisplay graphical information for a user interface on the input/outputdevice 1140.

The memory 1120 stores information within the system 1100. In oneimplementation, the memory 1120 is a computer-readable medium. In oneimplementation, the memory 1120 is a volatile memory unit. In anotherimplementation, the memory 1120 is a non-volatile memory unit.

The storage device 1130 is capable of providing mass storage for thesystem 400. In one implementation, the storage device 1130 is acomputer-readable medium. In various different implementations, thestorage device 1130 may be a floppy disk device, a hard disk device, anoptical disk device, or a tape device.

The input/output device 1140 provides input/output operations for thesystem 400. In one implementation, the input/output device 1140 includesa keyboard and/or pointing device. In another implementation, theinput/output device 1140 includes a display unit for displayinggraphical user interfaces.

The features described can be implemented in digital electroniccircuitry, or in computer hardware, firmware, software, or incombinations of them. The apparatus can be implemented in a computerprogram product tangibly embodied in an information carrier, e.g., in amachine-readable storage device for execution by a programmableprocessor; and method steps can be performed by a programmable processorexecuting a program of instructions to perform functions of thedescribed implementations by operating on input data and generatingoutput. The described features can be implemented advantageously in oneor more computer programs that are executable on a programmable systemincluding at least one programmable processor coupled to receive dataand instructions from, and to transmit data and instructions to, a datastorage system, at least one input device, and at least one outputdevice. A computer program is a set of instructions that can be used,directly or indirectly, in a computer to perform a certain activity orbring about a certain result. A computer program can be written in anyform of programming language, including compiled or interpretedlanguages, and it can be deployed in any form, including as astand-alone program or as a module, component, subroutine, or other unitsuitable for use in a computing environment.

Suitable processors for the execution of a program of instructionsinclude, by way of example, both general and special purposemicroprocessors, and the sole processor or one of multiple processors ofany kind of computer. Generally, a processor will receive instructionsand data from a read-only memory or a random access memory or both. Theessential elements of a computer are a processor for executinginstructions and one or more memories for storing instructions and data.Generally, a computer will also include, or be operatively coupled tocommunicate with, one or more mass storage devices for storing datafiles; such devices include magnetic disks, such as internal hard disksand removable disks; magneto-optical disks; and optical disks. Storagedevices suitable for tangibly embodying computer program instructionsand data include all forms of non-volatile memory, including by way ofexample semiconductor memory devices, such as EPROM, EEPROM, and flashmemory devices; magnetic disks such as internal hard disks and removabledisks; magneto-optical disks; and CD-ROM and DVD-ROM disks. Theprocessor and the memory can be supplemented by, or incorporated in,ASICs (application-specific integrated circuits).

To provide for interaction with a user, the features can be implementedon a computer having a display device such as a CRT (cathode ray tube)or LCD (liquid crystal display) monitor for displaying information tothe user and a keyboard and a pointing device such as a mouse or atrackball by which the user can provide input to the computer.Additionally, such activities can be implemented via touchscreenflat-panel displays and other appropriate mechanisms.

The features can be implemented in a computer system that includes aback-end component, such as a data server, or that includes a middlewarecomponent, such as an application server or an Internet server, or thatincludes a front-end component, such as a client computer having agraphical user interface or an Internet browser, or any combination ofthem. The components of the system can be connected by any form ormedium of digital data communication such as a communication network.Examples of communication networks include a local area network (“LAN”),a wide area network (“WAN”), peer-to-peer networks (having ad-hoc orstatic members), grid computing infrastructures, and the Internet.

The computer system can include clients and servers. A client and serverare generally remote from each other and typically interact through anetwork, such as the described one. The relationship of client andserver arises by virtue of computer programs running on the respectivecomputers and having a client-server relationship to each other.

While this specification contains many specific implementation details,these should not be construed as limitations on the scope of anyinventions or of what may be claimed, but rather as descriptions offeatures specific to particular implementations of particularinventions. Certain features that are described in this specification inthe context of separate implementations can also be implemented incombination in a single implementation. Conversely, various featuresthat are described in the context of a single implementation can also beimplemented in multiple implementations separately or in any suitablesubcombination. Moreover, although features may be described above asacting in certain combinations and even initially claimed as such, oneor more features from a claimed combination can in some cases be excisedfrom the combination, and the claimed combination may be directed to asubcombination or variation of a subcombination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products.

Thus, particular implementations of the subject matter have beendescribed. Other implementations are within the scope of the followingclaims. In some cases, the actions recited in the claims can beperformed in a different order and still achieve desirable results. Inaddition, the processes depicted in the accompanying figures do notnecessarily require the particular order shown, or sequential order, toachieve desirable results. In certain implementations, multitasking andparallel processing may be advantageous.

Pre-Recorded Sounds

Implementation 1 is a computer-implemented method. The method includesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece. The method furtherincludes recording, with the audio processing device and while playingthe source audio signal, an ambient audio signal that indicates soundspresent within an environment of the user. The method further includesidentifying, based on analysis of the ambient audio signal, that apre-recorded audio signal occurs in the ambient audio signal. The methodfurther includes accessing a second instance of the pre-recorded audiosignal. The method further includes modifying, with the audio processingdevice and using the stored instance of the pre-recorded audio signal asa representation of the pre-recorded audio signal that occurs in theambient audio signal, the source audio signal output by theelectroacoustic transducer of the user earpiece so as to at leastpartially remove the pre-recorded audio signal from the source audiosignal.

Implementation 2 is the method of implementation 1. The source audiosignal comprises an audio signal from at least one of a song or spokencontent.

Implementation 3 is the method of implementation 1. The pre-recordedaudio signal comprises an audio signal from at least one of a song,spoken content, a movie, or a television show.

Implementation 4 is the method of implementation 1. Identifying that thepre-recorded audio signal occurs in the ambient audio signal comprises:filtering, with the audio processing device, the ambient audio signal toextract the pre-recorded audio signal from the ambient audio signal; anddetermining that the pre-recorded audio signal extracted from theambient audio signal matches one of a plurality of differentpre-recorded audio signals.

Implementation 5 is the method of implementation 1. Identifying that thepre-recorded audio signal occurs in the ambient audio signal comprises:sending, from the audio processing device and over a network to a serverremotely located from the audio processing device, audio data thatcharacterizes at least a portion of the ambient audio signal; andreceiving, at the audio processing device as a response to the audiodata sent to the server, data that identifies the pre-recorded audiosignal.

Implementation 6 is the method of implementation 5. Accessing the storedinstance of the pre-recorded audio signal comprises receiving, at theaudio processing device as part of the response to the audio data sentto the server, the second instance of the pre-recorded audio signal.

Implementation 7 is the method of any one of implementations 1-6. Themethod includes determining a current temporal position of thepre-recorded audio signal that occurs in the ambient audio signal; andsynchronizing the second instance of the pre-recorded audio signal withthe pre-recorded audio signal that occurs in the ambient audio signalbased on the determined current temporal position.

Implementation 8 is the method of implementation 7. Modifying the sourceaudio signal so as to at least partially remove the pre-recorded audiosignal from the source audio signal comprises filtering the source audiosignal in real-time with the synchronized second instance of thepre-recorded audio signal.

Implementation 9 is the method of any one of implementations 1-8. Themethod comprises identifying that the pre-recorded audio signal nolonger occurs in the ambient audio signal; and in response toidentifying that the pre-recorded audio signal no longer occurs in theambient audio signal, ceasing modification of the source audio signal soas to at least partially remove the pre-recorded audio signal from thesource audio signal.

Implementation 10 is the method of implementation 9. The methodcomprises after ceasing modification of the source audio signal:identifying that the pre-recorded audio signal has resumed in theambient audio signal; and in response to identifying that thepre-recorded audio signal has resumed in the ambient audio signal,resuming modification of the source audio signal so as to at leastpartially remove the pre-recorded audio signal from the source audiosignal.

Implementation 11 is the method of any one of implementations 1-10.Recording the ambient audio signal comprises sensing the ambient audiosignal with the electroacoustic transducer of the user earpiece.

Implementation 12 is the method of any one of implementations 1-10.Recording the ambient audio signal comprises sensing the ambient audiosignal with a microphone that is external to the user earpiece.

Implementation 13 is the method of implementation 12. The microphone isfurther external to the audio processing device

Implementation 14 is the method of implementation 12. The microphone isan integrated component of the audio processing device.

Implementation 15 is the method of any one of implementations 1-14.Modifying the source audio signal output by the electroacoustictransducer of the user earpiece comprises subtracting the secondinstance of the pre-recorded audio signal from the source audio signal

Implementation 16 is the method of any one of implementations 1-15. Theearpiece comprises a headphone or an earbud.

Implementation 17 is the method of any one of implementations 1-16. Theaudio processing device comprises a portable digital media player, asmartphone, a tablet computing device, a notebook computing device, adesktop computing device, or a wearable computing device.

Implementation 18 is a computing system. The computing system includesone or more processors and one or more computer-readable media. The oneor more computer-readable media have instructions stored thereon that,when executed by the one or more processors, cause performance ofoperations that carry out any of the methods of implementations 1-17.

Implementation 19 is one or more computer-readable media that haveinstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out any of themethods of implementations 1-17.

Ear Presence

Implementation 1 is a computer-implemented method. The method includesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece; recording, with the audioprocessing device, an aural signal using the electroacoustic transducerof the user earpiece; determining values of one or more features of theaural signal that indicate, as a result of playing the source audiosignal, a characteristic of a space in which the user earpiece islocated; comparing the determined values of the one or more features ofthe aural signal with pre-defined values of the one or more features;and based on a result of comparing the determined values of the one ormore features of the aural signal with the pre-defined values of the oneor more features, determining whether the user earpiece is located at auser's ear.

Implementation 2 is the method of implementation 1. The method comprisesdetermining whether to perform a specified action based on whether theuser earpiece is located at the user's ear.

Implementation 3 is the method of implementation 1. The method comprisesdetermining, at a first time, that the user earpiece is located at theuser's ear; determining, at a second time after the first time, that thelocation of the user earpiece is changed from being at the user's ear tonot being at the user's ear; and in response to determining that thelocation of the user earpiece is changed from being at the user's ear tonot being at the user's ear, performing a specified action.

Implementation 4 is the method of implementation 1. The method comprisesdetermining, at a first time, that the user earpiece is not located atthe user's ear; determining, at a second time after the first time, thatthe location of the user earpiece is changed from being not at theuser's ear to being at the user's ear; and in response to determiningthat the location of the user earpiece is changed from being not at theuser's ear to being at the user's ear, performing a specified action.

Implementation 5 is the method of any one of implementations 2-4. Thespecified action comprises at least one of playing a media file, pausingthe media file, stopping the media file, resuming play of a paused mediafile, activating an electronic display, adjusting a brightness settingof a backlight of the electronic display, transitioning a device into alocked mode, transitioning a device out of a locked mode, initiating atelephone call, ending a telephone call, launching an application, orclosing an application.

Implementation 6 is the method of any one of implementations 1-5. Themethod comprises repeatedly determining whether the user earpiece islocated at the user's ear over a period of time to monitor for a changein location of the user earpiece from (i) not being at the user's ear tobeing at the user's ear or (ii) from being at the user's ear to notbeing at the user's ear.

Implementation 7 is the method of any one of implementations 1-6. Themethod comprises recording the aural signal with the audio processingdevice concurrently with playing the source audio signal with the audioprocessing device.

Implementation 8 is the method of any one of implementation 1-6. Theelectroacoustic transducer of the user earpiece concurrently functionsas (i) a speaker that audibly outputs the source audio signal and (ii) amicrophone that senses the aural signal while the source audio signal isplayed.

Implementation 9 is the method of any one of implementations 1-8. Theone or more features of the aural signal comprise an impulse response ofan acoustic echo of the space in which the user earpiece is located.

Implementation 10 is the method of any one of implementations 1-9. Thepre-defined values of the one or more features comprise a model thatcharacterizes a space in the user's ear.

Implementation 11 is the method of any one of implementations 1-10. Thepre-defined values of the one or more features are determined based atleast in part on previously determined values of the one or morefeatures of aural signals that resulted from playing the source audiosignal or another audio signal when the user earpiece was determined tobe located at the user's ear.

Implementation 12 is the method of any one of implementations 1-10. Thepre-defined values of the one or more features are determined based atleast in part on values of the one or more features of aural signalsthat resulted from playing the source audio signal or another audiosignal when the user earpiece was located at the respective ears of oneor more users other than the user.

Implementation 13 is the method of any one of implementations 1-12. Theearpiece comprises a headphone or an earbud.

Implementation 14 is the method of any one of implementations 1-13. Theaudio processing device comprises a portable digital media player, asmartphone, a tablet computing device, a notebook computing device, adesktop computing device, or a wearable computing device.

Implementation 15 is the method of any one of implementations 1-14.Playing the source audio signal comprises playing white noise.

Implementation 16 is the method of any one of implementations 1-14.Playing the source audio signal comprises playing a song or spokencontent to the user of the audio processing device.

Implementation 17 is the method of any one of implementations 1-16.Playing the source audio signal comprises playing sound having anaverage frequency that is greater than 20 kiloHertz, such that theplayed sound is above a normal frequency limit for human hearing.

Implementation 18 is a computing system. The computing system includesone or more processors and one or more computer-readable media. The oneor more computer-readable media have instructions stored thereon that,when executed by the one or more processors, cause performance ofoperations that carry out any of the methods of implementations 1-17.

Implementation 19 is one or more computer-readable media that haveinstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out any of themethods of implementations 1-17.

User Authentication

Implementation 1 is a computer-implemented method. The method includesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece. The method furtherincludes recording, with the audio processing device, an aural signalthat is sensed by the electroacoustic transducer of the user earpiece.The method further includes determining values of one or more featuresof the aural signal that indicate, as a result of playing the sourceaudio signal, a characteristic of a space in which the user earpiece islocated. The method further includes generating an acoustic signaturefor the user based on the values of the one or more features of theaural signal. The method further includes registering the acousticsignature with a user account.

Implementation 2 is the method of implementation 1. The method furtherincludes comprising after registering the acoustic signature with theuser account: playing, with the audio processing device, a second sourceaudio signal, including causing the second source audio signal to beaudibly output by the electroacoustic transducer of the user earpiece;recording, with the audio processing device, a second aural signal thatis sensed by the electroacoustic transducer of the user earpiece;determining values of the one or more features of the second auralsignal; determining whether the one or more features of the second auralsignal match the acoustic signature that is registered with the useraccount; and determining, based on whether the one or more features ofthe second aural signal match the acoustic signature that is registeredwith the user account, whether to perform a specified action.

Implementation 3 is the method of implementation 2. Determining whetherto perform the specified action comprises authenticating a user of theuser earpiece based on whether the one or more features of the secondaural signal match the acoustic signature that is registered with theuser account.

Implementation 4 is the method of implementation 2. The specified actioncomprises logging into the user account or another account.

Implementation 5 is the method of implementation 2. The method includesdetermining that the one or more features of the second aural signalmatch the acoustic signature that is registered with the user account;and in response to determining that the one or more features of thesecond aural signal match the acoustic signature that is registered withthe user account, permitting performance of the specified action.

Implementation 6 is the method of implementation 2. The method includesdetermining that the one or more features of the second aural signal donot match the acoustic signature that is registered with the useraccount; and in response to determining that the one or more features ofthe second aural signal do not match the acoustic signature that isregistered with the user account, blocking performance of the specifiedaction.

Implementation 7 is the method of implementation 2. The method includesrepeatedly determining values of the one or more features of the secondaural signal while the audio processing device is in a first modeassociated with the specified action; determining to remain in the firstmode of the audio processing device associated with the specified actionso long as the values of the one or more features of the second auralsignal determined in iterations of the repeatedly determining match theacoustic signature that is registered with the user account.

Implementation 8 is the method of implementation 7. The method furtherincludes determining to transition from the first mode associated withthe specified action to a second mode associated with the specifiedaction in response to determining that the values of the one or morefeatures of the second aural signal no longer match the acousticsignature that is registered with the user account.

Implementation 9 is the method of implementation 7. The first modeassociated with the specified action comprises being logged into arestricted user account.

Implementation 10 is the method of implementation 8. The first modeassociated with the specified action comprises being logged into arestricted user account, wherein the second mode associated with thespecified action comprises being logged out of the restricted useraccount.

Implementation 11 is the method of any one of implementations 1-10. Themethod comprises correlating the acoustic signature with a firstposition of the user earpiece at the user's ear.

Implementation 12 is the method of any one of implementations 1-11. Themethod comprises performing multiple iterations of the steps of playing,recording, determining, and generating so as to generate multipleacoustic signatures for the user; prompting the user to move the userearpiece to a different position at the user's ear for each iteration ofthe multiple iterations; and correlating at least two of the multipleacoustic signatures for the user with at least two different positionsof the user earpiece at the user's ear.

Implementation 13 is the method of any one of implementations 1-12. Themethod includes recording the aural signal with the audio processingdevice concurrently with playing the source audio signal with the audioprocessing device.

Implementation 14 is the method of any one of implementations 1-13. Theelectroacoustic transducer of the user earpiece concurrently functionsas (i) a speaker that audibly outputs the source audio signal and (ii) amicrophone that senses the aural signal while the source audio signal isplayed.

Implementation 15 is the method of any one of implementations 1-14. Theone or more features of the aural signal comprise an impulse response ofan acoustic echo of the space in which the user earpiece is located.

Implementation 16 is the method of any one of implementations 1-15. Thepre-defined values of the one or more features comprise a model thatcharacterizes a space in the user's ear.

Implementation 17 is the method of any one of implementations 1-16. Theuser earpiece comprises a headphone or an earbud.

Implementation 18 is the method of any one of implementations 1-17.Playing the source audio signal comprises playing white noise.

Implementation 19 is the method of any one of implementations 1-17.Playing the source audio signal comprises playing a song or spokencontent to the user of the audio processing device.

Implementation 20 is the method of any one of implementations 1-17.Playing the source audio signal comprises playing sound having anaverage frequency that is greater than 20 kiloHertz, such that theplayed sound is above a normal frequency limit for human hearing.

Implementation 21 is a computing system. The computing system includesone or more processors and one or more computer-readable media. The oneor more computer-readable media have instructions stored thereon that,when executed by the one or more processors, cause performance ofoperations that carry out the methods of implementations 1-20.

Implementation 22 is one or more computer-readable media havinginstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out the methodsof any of implementations 1-20.

Using Acoustic Signatures

Implementation 1 is a computer-implemented method. The method includesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece. The method furtherincludes recording, with the audio processing device, an aural signalthat is sensed by the electroacoustic transducer of the user earpiece.The method further includes determining values of one or more featuresof the aural signal that indicate, as a result of playing the sourceaudio signal, a characteristic of a space in which the user earpiece islocated. The method further includes determining whether the one or morefeatures of the aural signal match one or more acoustic signatures thatare registered with a user account. The method further includesdetermining, based on whether the one or more features of the auralsignal match the one or more acoustic signatures that are registeredwith the user account, whether to perform a specified action.

Implementation 2 is a computing system. The computing system includesone or more processors and one or more computer-readable media. The oneor more computer-readable media have instructions stored thereon that,when executed by the one or more processors, cause performance ofoperations that carry out the method of implementation 1.

Implementation 3 is one or more computer-readable media havinginstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out the method ofimplementation 1.

Configuring Simultaneous Playing and Recording

Implementation 1 is a system to simultaneously play and record audiousing a same transducer. The system includes an audio processing circuitthat

includes an audio output connection, an audio input connection, and aground connection. The system includes a transducer that includes afirst electrical connection and a second electrical connection. Thesystem includes a resistor that is connected between the audio outputconnection of the audio processing circuit and the audio inputconnection of the audio processing circuit. The system includes a firstelectrical conductor between the first electrical connection of thetransducer and either the audio output connection of the audioprocessing circuit or the audio input connection of the audio processingcircuit. The system includes a second electrical conductor between thesecond electrical connection of the transducer and the ground connectionof the audio processing circuit. Implementation 2 is the system ofimplementation 1, wherein the first electrical conductor is a wire, andthe second electrical conductor is a wire.

Implementation 3 is the system of implementation 1, wherein: aresistance between the first electrical connection of the transducer andthe audio output connection of the audio processing circuit is less than1 ohm, or the resistance between the first electrical connection of thetransducer and the audio input connection of the audio processingcircuit is less than 1 ohm.

Implementation 4 is the system of implementation 1, wherein the resistorhas a resistance greater than 5 ohms.

Implementation 5 is the system of implementation 1, wherein the resistorhas a resistance greater than 50 ohms.

Implementation 6 is the system of implementation 1, further comprising afirst circuit element that is configured to switch the resistor that isconnected between the audio output connection and the audio inputconnection so that the resistor is no longer connected between the audiooutput connection and the audio input connection.

Implementation 7 is the system of implementation 6, comprising the firstcircuit element or a second circuit element that is configured toconnect the first electrical connection of the transducer to the audiooutput connection of the transducer or the audio input connection of thetransducer as a result of the resistor being switched so that it is nolonger connected between the audio output connection and the audio inputconnection.

Implementation 8 is the system of implementation 1, wherein the firstelectrical conductor is connected between the first electricalconnection of the transducer and the audio input connection of the audioprocessing circuit

Implementation 9 is the system of implementation 1, wherein the audioinput connection includes a first input connection that is connected toa first side of the resistor and a second input connection that isconnected to a second side of the resistor, such that the first inputconnection and the second input connection are connected across theresistor.

Temperature Measurement Mechanism

Implementation 1 is a computer-implemented method. The method comprisesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece. The method comprisesrecording, with the audio processing device and while playing the sourceaudio signal, a recorded audio signal using the electroacoustictransducer of the user earpiece. The method comprises identifying, bythe audio processing device, one or more parameters that indicate howproperties of the user earpiece affect playing of the source audiosignal by the electroacoustic transducer, wherein at least one of theone or more parameters is temperature dependent. The method comprisesdetermining, by the audio processing device, a temperature value that isestimated to cause the source audio signal that was played by the audioprocessing device to result in the recorded audio signal, accounting forchanges to the source audio signal that occur due to application of theone or more parameters.

Implementation 2 is the computer-implemented method of implementation 1,wherein the at least one of the one or more parameters is atemperature-dependent impulse response of the user earpiece.

Implementation 3 is the computer-implemented method of implementation 1,wherein the at least one of the one or more parameters is atemperature-dependent frequency response of the user earpiece.

Implementation 4 is the computer-implemented method of implementation 1,wherein the temperature value represents a relative temperature value ofthe transducer. The method further comprises modifying the temperaturevalue using an offset constant and a scaling constant to generate anabsolute temperature value of the user earpiece, wherein the offsetconstant and the scaling constant are values that were calibrated forthe user earpiece or a type of the user earpiece.

Implementation 5 is the computer-implemented method of implementation 1,wherein the temperature value represents a temperature value of thetransducer. The method further comprises estimating, using thedetermined temperature value, an environmental temperature value of anenvironment in which the user earpiece is located by identifying atemperature of the user earpiece at a future time using a model thatrepresents earpiece temperature-change characteristics over time.

Implementation 6 is the computer-implemented method of implementation 1,wherein the temperature value represents a temperature value of thetransducer. The method further comprises estimating, using thedetermined temperature value and an environmental temperature value ofan environment in which the user earpiece is located, a user bodytemperature value of a user body to which the user earpiece is adjacentand touching by correlating the user body temperature value topreviously-calibrated sets of user body temperature, user earpiecetemperature, and environmental temperature that correlate to each other.

Implementation 7 is the computer-implemented method of implementation 1.The method further comprises determining, by the audio processingdevice, that the user earpiece is not located in a user ear, wherein theplaying of the source audio signal and the recording of the recordedaudio signal occur while the audio processing device has determined thatthe user earpiece is not located in the user ear, such that thedetermined temperature value is designated as an environmentaltemperature value.

Implementation 8 is the computer-implemented method of implementation 7.The method further comprises determining, by the audio processing deviceand after the user earpiece has been determined to not be located in theuser ear, that the user earpiece is now located in a user ear, and inresponse: (i) playing, with the audio processing device, a second sourceaudio signal using the electroacoustic transducer, (ii) recording, withthe audio processing device and while playing the second source audiosignal, a second recorded audio signal, and (iii) determining, by theaudio processing device, a second temperature value that is estimated tocause the second source audio signal that was played by the audioprocessing device to result in the second recorded audio signal,accounting for changes to the second source audio signal that occur dueto application of one or more parameters, wherein the second temperaturevalue differs from the determined temperature value that is designatedas the environmental temperature value because temperature of theearpiece has changed after the earpiece was placed in the user ear. Themethod further comprises estimating, by the audio processing device, auser body temperature value using the determined temperature value thatis designated as the environmental temperature value and the secondtemperature value.

Implementation 9 is the computer-implemented method of implementation 8,wherein estimating the user body temperature using the determinedtemperature value and the second temperature value includes correlatingthe user body temperature to previously-calibrated sets of user bodytemperature, user earpiece temperature, and environmental temperaturethat correlate to each other.

Implementation 10 is a computing system comprising: one or moreprocessors; and one or more computer-readable media having instructionsstored thereon that, when executed by the one or more processors, causeperformance of operations that carry out any of the method of claims1-9.

Implementation 11 is one or more computer-readable media havinginstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out any of themethods of claims 1-9.

Another Temperature Measurement Mechanism

Implementation 1 is a computer-implemented method. The method comprisesplaying, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece. The method comprisesrecording, with the audio processing device and while playing the sourceaudio signal, a recorded audio signal using the electroacoustictransducer of the user earpiece. The method comprises identifying, bythe audio processing device, one or more parameters that indicate howproperties of the earpiece affect playing of the source audio signal bythe electroacoustic transducer. The method comprises determining, by theaudio processing device, a parameter that indicates how properties ofthe user ear environment at which the user earpiece is located affectsthe source audio signal that was played by the audio processing device,accounting for changes to the source audio signal that occur due toapplication of the one or more parameters that indicate how propertiesof the earpiece affect playing of the source audio signal. The methodcomprises using, by the audio processing device, the parameter thatindicates how the properties of the user ear affects the source audiosignal to determine a temperature value.

Implementation 2 is the computer-implemented method of implementation 1,wherein the temperature value represents a relative temperature value ofthe user ear environment. The method further comprises modifying thetemperature value using an offset constant and a scaling constant togenerate an absolute temperature value of the user ear environment,wherein the offset constant and the scaling constant are values thatwere calibrated for the user earpiece or a type of the user earpiece.

Implementation 3 is a computing system comprising one or moreprocessors; and one or more computer-readable media having instructionsstored thereon that, when executed by the one or more processors, causeperformance of operations that carry out any of the method ofimplementations 1-2.

Implementation 4 is one or more computer-readable media havinginstructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that carry out any of themethods of implementations 1-2.

In some implementations, the audio processing device may be configuredto perform various combinations of the techniques described herein,including two or more of the in-ear detection techniques, active-noisecancellation with pre-recorded sounds techniques, and aural-basedauthentication techniques.

In some implementations, the techniques described herein may beperformed using wireless earpieces that are communicably coupled with acomputing device via a wireless connection (e.g., BLUETOOTH). Forexample, the wireless earpieces may digitally sample recorded sounds andeither process the sounds locally at the headset or transmit therecorded signal to a separate audio processing device to perform, e.g.,aural-based authentication, in-ear detection, and/or active-noisecancellation with pre-recorded sounds.

What is claimed is:
 1. A computer-implemented method, comprising:playing, with an audio processing device, a source audio signal,including causing the source audio signal to be audibly output by anelectroacoustic transducer of a user earpiece; recording, with the audioprocessing device, an aural signal using same said electroacoustictransducer of the user earpiece; determining values of one or morefeatures of the aural signal that indicate, as a result of playing thesource audio signal, a characteristic of a space in which the userearpiece is located; comparing the determined values of the one or morefeatures of the aural signal with defined values; and based on a resultof comparing the determined values of the one or more features of theaural signal with the defined values, determining whether the userearpiece is located at a user's ear.
 2. The computer-implemented methodof claim 1, further comprising determining whether to perform aspecified action based on whether the user earpiece is located at theuser's ear.
 3. The computer-implemented method of claim 1, furthercomprising: determining, at a first time, that the user earpiece islocated at the user's ear; determining, at a second time after the firsttime, that the location of the user earpiece is changed from being atthe user's ear to not being at the user's ear; and in response todetermining that the location of the user earpiece is changed from beingat the user's ear to not being at the user's ear, performing a specifiedaction.
 4. The computer-implemented method of claim 3, wherein thespecified action comprises at least one of playing a media file, pausingthe media file, stopping the media file, resuming play of a paused mediafile, activating an electronic display, adjusting a brightness settingof a backlight of the electronic display, transitioning a device into alocked mode, transitioning a device out of a locked mode, initiating atelephone call, ending a telephone call, launching an application, orclosing an application.
 5. The computer-implemented method of claim 1,further comprising: determining, at a first time, that the user earpieceis not located at the user's ear; determining, at a second time afterthe first time, that the location of the user earpiece is changed frombeing not at the user's ear to being at the user's ear; and in responseto determining that the location of the user earpiece is changed frombeing not at the user's ear to being at the user's ear, performing aspecified action.
 6. The computer-implemented method of claim 5, whereinthe specified action comprises at least one of playing a media file,pausing the media file, stopping the media file, resuming play of apaused media file, activating an electronic display, adjusting abrightness setting of a backlight of the electronic display,transitioning a device into a locked mode, transitioning a device out ofa locked mode, initiating a telephone call, ending a telephone call,launching an application, or closing an application.
 7. Thecomputer-implemented method of claim 1, further comprising repeatedlydetermining whether the user earpiece is located at the user's ear overa period of time to monitor for a change in location of the userearpiece from (i) not being at the user's ear to being at the user's earor (ii) from being at the user's ear to not being at the user's ear. 8.The computer-implemented method of claim 1, further comprising recordingthe aural signal with the audio processing device concurrently withplaying the source audio signal with the audio processing device.
 9. Thecomputer-implemented method of claim 1, wherein same saidelectroacoustic transducer of the user earpiece concurrently functionsas (i) a speaker that audibly outputs the source audio signal and (ii) amicrophone that senses the aural signal while the source audio signal isplayed.
 10. The computer-implemented method of claim 1, wherein the oneor more features of the aural signal comprise an impulse response of anacoustic echo of the space in which the user earpiece is located. 11.The computer-implemented method of claim 1, wherein the defined valuescomprise a model that characterizes a space in the user's ear.
 12. Thecomputer-implemented method of claim 1, wherein the defined values aredetermined based at least in part on previously determined values of theone or more features of aural signals that resulted from playing thesource audio signal or another audio signal when the user earpiece wasdetermined to be located at the user's ear.
 13. The computer-implementedmethod of claim 1, wherein the defined values are determined based atleast in part on values of the one or more features of aural signalsthat resulted from playing the source audio signal or another audiosignal when the user earpiece was located at the respective ears of oneor more users other than the user.
 14. The computer-implemented methodof claim 1, wherein the earpiece comprises a headphone or an earbud. 15.The computer-implemented method of claim 1, wherein the audio processingdevice comprises a portable digital media player, a smartphone, a tabletcomputing device, a notebook computing device, a desktop computingdevice, or a wearable computing device.
 16. The computer-implementedmethod of claim 1, wherein playing the source audio signal comprisesplaying white noise.
 17. The computer-implemented method of claim 1,wherein playing the source audio signal comprises playing a song orspoken content to the user of the audio processing device.
 18. Thecomputer-implemented method of claim 1, wherein playing the source audiosignal comprises playing sound having an average frequency that isgreater than 20 kilohertz, such that the played sound is above a normalfrequency limit for human hearing.
 19. A computing system, comprising:one or more processors; and one or more computer-readable media havinginstructions stored thereon that, when executed by the one or moreprocessors, cause performance of operations that comprise: playing, withan audio processing device, a source audio signal, including causing thesource audio signal to be audibly output by an electroacoustictransducer of a user earpiece; recording, with the audio processingdevice, an aural signal using same said electroacoustic transducer ofthe user earpiece; determining values of one or more features of theaural signal that indicate, as a result of playing the source audiosignal, a characteristic of a space in which the user earpiece islocated; comparing the determined values of the one or more features ofthe aural signal with defined values; and based on a result of comparingthe determined values of the one or more features of the aural signalwith the defined values, determining whether the user earpiece islocated at a user's ear.
 20. One or more computer-readable media thathave instructions stored thereon that, when executed by one or moreprocessors, cause performance of operations that comprise: playing, withan audio processing device, a source audio signal, including causing thesource audio signal to be audibly output by an electroacoustictransducer of a user earpiece; recording, with the audio processingdevice, an aural signal using same said electroacoustic transducer ofthe user earpiece; determining values of one or more features of theaural signal that indicate, as a result of playing the source audiosignal, a characteristic of a space in which the user earpiece islocated; comparing the determined values of the one or more features ofthe aural signal with defined values; and based on a result of comparingthe determined values of the one or more features of the aural signalwith the defined values, determining whether the user earpiece islocated at a user's ear.